April 16, 2012

Python based Technology Stack at Votizen

Category: Software development,Technology,Work — by Amit Chaudhary @ 6:52 pm

Deep details of Python based Technology Stack at Votizen from the Inside Votizen post by Paul Stamatiou.

  • Amazon Web Services
    • webservers – 2 large instances running Apache, Django, Python
    • load balancer – 1 large instance running HAProxy and Stunnel
    • asynchronous workers – 3 x-large instances running several hundred Celery workers each
    • reverse geocoding & districts – 3 x-large instances running what we call politicoder
    • elastic search for voters – 2 High-Memory Quadruple Extra Large instance running elastic search with entire corpus of voters loaded into memory (verify 50-75 people/second)
    • error reporting – 1 large instance running a sentry server
    • utility server – 1 large instance used for various opsy operations
    • db-misc server – 1 large instance running redis, memcached, and RabbitMQ
    • admin server – 1 large instance used for managing deployment and monitoring site, runs CruiseControl (for deployment) and Nagios (for monitoring)
  • Puppet\Chef - We were previously using Puppet to manage server configurations in our admin repository, but are in the process of moving to Chef, as our new ops person prefers Chef. Both are Ruby-based tools, but are more mature than any comparable Python solution
  • Python - We write almost all our code in Python 2.6, because Python gives you wings
  • Django - This is one of the most fully-baked Python web frameworks and we have 2 core contributors on our team. Version 1.4 was just released, but we are still running on 1.3
  • Redis - We use redis mostly to handle caching of large lists of ids, because it has really good set/list operations built into it.
  • Celery - We have to do a log of user processing after they sign up and even when they sign in: fetch contacts on your social network, find your contacts districts and see if they are voters, find your districts and see if you are a voter
  • RabbitMQ - The default message queue used by Celery, and also just a good queuing system. We only use it for celery.
  • Memcached - Cache whatever you can, whenever you can. Django has good memcached support and we try to take advantage of it as much as possible.

We have unit testing and continuous integration, there’s no staging environment, everything goes straight to production. We have feature flagging now to limit development features to staff.

We’re looking to move to Vagrant so the development and production environments can be nearly identical.

• • •

March 10, 2012

RFS 10: Design and Simplicity, a Request for Startup article

Category: Entrepreneur — by Amit Chaudhary @ 11:24 pm

RFS 10: Design and Simplicity

I wrote an article in the spirit of YCombinator’s ‘s Request for Startup series.

It was inspired by the PyCon 2012 keynote by Paul Graham of YCombinator.
The talk is covered in the article Frighteningly Ambitious Startup Ideas.

• • •

January 8, 2012

2012 Python Meme from Amit: HeapQ, Flask, Python on Quora & Python 3

Category: Software development — by Amit Chaudhary @ 4:18 pm
  1. What is the coolest Python application, framework, or library you
    have discovered in 2011?
  2. What new programming technique did you learn in 2011?
  3. What’s the name of the open source project you contributed the
    most in 2011? What did you do?
    • Nothing in particular
  4. What was the Python blog or website you read the most in 2011?
  5. What are the three top things you want to learn in 2012?
    • At the top is more public projects on github & else where.
    • Let doing (public projects) decide what I learn
    • SQLAlChemy
  6. What are the top software, app, or lib you wish someone would write in 2012?

Tarek started this series with these instructions:

  • copy-paste the questions and answer to them in your blog
  • tweet it with the #2012pythonmeme hashtag
• • •

December 22, 2011

Top TEDx Conferences courtesy Youtube API & Python

Category: Life,Personal development,Software development — by Amit Chaudhary @ 5:17 pm

For some good conferences, these were the conferences with most talks in the top 500 talks, the number of talks & video link to top talk from that conference.

TEDxTokyo,24

TEDxBuenosAires,18

TEDxCairo,17

TEDxParis,17

TEDxDubai,14

TEDxCaltech,12

TEDxNYED,10

TEDxEast,7

TEDxLahore,7

TEDxTaipei,7

I wrote couple of Python (programming language) scripts that used YouTube API. The approach, find TEDx conference with a larger number of high quality talks, the quality measured by view count.

Some additional notes:
-A check in top 200, brings in TEDxBerlin, TEDxCMU, TEDxNASA & TEDxToronto
-A check in top 1000 brings in TEDxSF, TEDxAmsterdam & TEDxAcademy
-The code is bunch of small Python scripts and is in github at https://github.com/amitch/ytube-tedx

• • •

August 11, 2011

On Quora: Workspace, Problem Solving, Tools for Python Developer, Real time in Python or Ruby

Category: Personal development,Software development,Work — by Amit Chaudhary @ 11:01 am
Rooms with different engineers followed by Open space are the best among the other choices. It
-Encourages collaborationidea flow & camaraderie.
-promotes visibility & transparency.
-is efficient in terms of cost & time to setup.

The biggest drawback is that it does not leave space (mental, sound & time kind of space) for flow to happen easily. Hence you will see productive programming start in late afternoon or lots of head phones. This is less so in shared rooms than in Open space.

Personally I prefer silence during work and have worked in startups with Cubicles, Office(great view of Mt. Rainier), Shared rooms & Open space and would rate individual offices as worse of the lot due to the walls, real and virtual.

How does one become good at problem solving?

Work at becoming a better solver in addition to solving new problems.
See Topcoder tutorials such as How to find solution and how to dissect a problem.
http://www.topcoder.com/tc?modul…

Use Charlie Munger‘s “Mental models” method as another approach, they are covered in this pdf http://www.focusinvestor.com/Foc… & better in his book, Poor Charlie’s Almanack, http://www.poorcharliesalmanack….

What tools should a modern Python developer have under his belt?

I would like to propose a 6-12 month list, instead of a 5 year list.

Find out & use
-details of python REPL including dir & help command

-the Python coding standards, PEP 8 & Python API documentation standard, PEP 257.

-Read pythonic code, options include Guido’s essays likehttp://www.python.org/doc/essays… or Dare Obasanjo’s python code and followup posts,
http://www.google.com/search?hl=…

-Code to write: Runtime object alteration, even if you do not use it latter. Write an iterator.

-the Python standard library including string, date, persistence, regex, etc.

-create a python package & upload to pypi.

-write some unit tests with unittest\PyUnit\nosehttp://docs.python.org/library/u…

For using Python, consider Tornado (web framework) by the FriendFeed (nowFacebook Inc. (company)) team.

Go through their documentation & excellent demos, https://github.com/facebook/torn….
If you prefer a book, go through “Building the real time user experience” by Ted Roden

For using Ruby, you can use what VMware Cloud Foundry Platform-as-a-Service (PaaS) uses, Thin webserver, Ruby 1.9 fibers and async DB drivers, see this article http://www.igvita.com/2011/04/14…

For Ruby I have also seen references to orbited & juggernaut.

• • •

August 19, 2010

Entrepreneurship Success Factors, Before & After, Highlights from Survey by Vivek Wadhwa & others

Category: Entrepreneur — by Amit Chaudhary @ 6:18 pm

Following are highlights from Survey and Study by Vivek Wadhwa & others: Anatomy of an Entrepreneur Part II:Making of A Successful Entrepreneur

This research is based on a survey of 549 company founders in a variety of industries, including aerospace and defense, computer and electronics, health care,
and services.

Entrepreneurs Obstacles from their past, Time and effort required, capital/financing, and experience in running a business
We asked company founders to rank the challenges they faced in starting their businesses.

• 61 percent said amount of time and effort required was a challenge

• 52 percent said lack of available capital/financing was a challenge

• 52 percent said lack of prior experience in running a business was a challenge

Machu Pichu, Peru

Entrepreneurs Obstacles for others, Lack of willingness to take risks, Time and effort required, Raising capital, Business management skills and Family or financial pressure

• The factor most commonly ranked as important—by 98 percent—was lack of willingness or of ability to take risks, with 50 percent believing this to be an extremely important barrier to entrepreneurship. This clearly indicates that these company founders considered entrepreneurship to be a risky endeavor.

• 93 percent said that the amount of time and effort required was an important barrier.

• 91 percent said that difficulty in raising capital was an important inhibitor.

• 89 percent said business management skills, 84 percent said knowledge of how to start a business, and 83 percent said knowledge about the industry and markets were important issues.

• 73 percent believed that family or financial pressures to keep a traditional, steady job were issues.

Experience, Management, and Luck: The Keys to Success 

• 96 percent ranked prior work experience as an important success factor

Professional Networks, Education, Funding, Personal Networks: Important

• 73 percent said professional networks were important to the success of their current businesses.

Location, Investor Advice, Alumni Networks, and Regional Assistance: Not so Important 

• Entrepreneurs were almost evenly divided about the importance of the location of their businesses. 50 percent said location was important.

Using Personal Savings is the Norm, Venture Capital Comes to the Experienced, and Friends and Family are Always There

• The most significant source of funding for all businesses was company founders’ personal savings: 70 percent said they had used personal savings as a main source of funding for their first business, more than four times the number chiefly financed by any other type of funding.

• • •
Next Page »
Powered by: WordPress Theme based on Sharepoint like theme from: ADMIN-BG