April 16, 2012

Python based Technology Stack at Votizen

Category: Software development,Technology,Work — by Amit Chaudhary @ 6:52 pm

Deep details of Python based Technology Stack at Votizen from the Inside Votizen post by Paul Stamatiou.

  • Amazon Web Services
    • webservers – 2 large instances running Apache, Django, Python
    • load balancer – 1 large instance running HAProxy and Stunnel
    • asynchronous workers – 3 x-large instances running several hundred Celery workers each
    • reverse geocoding & districts – 3 x-large instances running what we call politicoder
    • elastic search for voters – 2 High-Memory Quadruple Extra Large instance running elastic search with entire corpus of voters loaded into memory (verify 50-75 people/second)
    • error reporting – 1 large instance running a sentry server
    • utility server – 1 large instance used for various opsy operations
    • db-misc server – 1 large instance running redis, memcached, and RabbitMQ
    • admin server – 1 large instance used for managing deployment and monitoring site, runs CruiseControl (for deployment) and Nagios (for monitoring)
  • Puppet\Chef - We were previously using Puppet to manage server configurations in our admin repository, but are in the process of moving to Chef, as our new ops person prefers Chef. Both are Ruby-based tools, but are more mature than any comparable Python solution
  • Python - We write almost all our code in Python 2.6, because Python gives you wings
  • Django - This is one of the most fully-baked Python web frameworks and we have 2 core contributors on our team. Version 1.4 was just released, but we are still running on 1.3
  • Redis - We use redis mostly to handle caching of large lists of ids, because it has really good set/list operations built into it.
  • Celery - We have to do a log of user processing after they sign up and even when they sign in: fetch contacts on your social network, find your contacts districts and see if they are voters, find your districts and see if you are a voter
  • RabbitMQ - The default message queue used by Celery, and also just a good queuing system. We only use it for celery.
  • Memcached - Cache whatever you can, whenever you can. Django has good memcached support and we try to take advantage of it as much as possible.

We have unit testing and continuous integration, there’s no staging environment, everything goes straight to production. We have feature flagging now to limit development features to staff.

We’re looking to move to Vagrant so the development and production environments can be nearly identical.

• • •

March 10, 2012

RFS 10: Design and Simplicity, a Request for Startup article

Category: Entrepreneur — by Amit Chaudhary @ 11:24 pm

RFS 10: Design and Simplicity

I wrote an article in the spirit of YCombinator’s ‘s Request for Startup series.

It was inspired by the PyCon 2012 keynote by Paul Graham of YCombinator.
The talk is covered in the article Frighteningly Ambitious Startup Ideas.

• • •

January 8, 2012

2012 Python Meme from Amit: HeapQ, Flask, Python on Quora & Python 3

Category: Software development — by Amit Chaudhary @ 4:18 pm
  1. What is the coolest Python application, framework, or library you
    have discovered in 2011?
  2. What new programming technique did you learn in 2011?
  3. What’s the name of the open source project you contributed the
    most in 2011? What did you do?
    • Nothing in particular
  4. What was the Python blog or website you read the most in 2011?
  5. What are the three top things you want to learn in 2012?
    • At the top is more public projects on github & else where.
    • Let doing (public projects) decide what I learn
    • SQLAlChemy
  6. What are the top software, app, or lib you wish someone would write in 2012?

Tarek started this series with these instructions:

  • copy-paste the questions and answer to them in your blog
  • tweet it with the #2012pythonmeme hashtag
• • •

December 22, 2011

Top TEDx Conferences courtesy Youtube API & Python

Category: Life,Personal development,Software development — by Amit Chaudhary @ 5:17 pm

For some good conferences, these were the conferences with most talks in the top 500 talks, the number of talks & video link to top talk from that conference.











I wrote couple of Python (programming language) scripts that used YouTube API. The approach, find TEDx conference with a larger number of high quality talks, the quality measured by view count.

Some additional notes:
-A check in top 200, brings in TEDxBerlin, TEDxCMU, TEDxNASA & TEDxToronto
-A check in top 1000 brings in TEDxSF, TEDxAmsterdam & TEDxAcademy
-The code is bunch of small Python scripts and is in github at https://github.com/amitch/ytube-tedx

• • •

December 19, 2011

100 life things to do, 2011 update

Category: Life — by Amit Chaudhary @ 11:15 pm

Three years I posted a list about 100 life things to do, I noticed some changes have happened since then.

bold are done since 2011 and underlined are worth a try in the future.

2. Slept under the stars

5. Watched a meteor shower

11. Bungee jumped

12. Visited Paris

17. Walked to the top of the Statue of Liberty

22. Hitch hiked

27. Run a Marathon

28. Ridden in a gondola in Venice

29. Seen a total eclipse

32. Been on a cruise

37. Had enough money to be truly satisfied

43. Bought a stranger a meal at a restaurant

50. Been to the top of the Eiffel Tower in Paris

51. Gone scuba diving or snorkeling

53. Played in the mud

57. Started a business

58. Taken a martial arts class

60. Served at a soup kitchen

65. Gone sky diving

68. Flown in a helicopter

74. Toured the Everglades

86. Visited the White House

89. Saved someone’s life


Done: 4/26

Worth doing someday: 18/26

• • •

August 11, 2011

On Quora: Workspace, Problem Solving, Tools for Python Developer, Real time in Python or Ruby

Category: Personal development,Software development,Work — by Amit Chaudhary @ 11:01 am
Rooms with different engineers followed by Open space are the best among the other choices. It
-Encourages collaborationidea flow & camaraderie.
-promotes visibility & transparency.
-is efficient in terms of cost & time to setup.

The biggest drawback is that it does not leave space (mental, sound & time kind of space) for flow to happen easily. Hence you will see productive programming start in late afternoon or lots of head phones. This is less so in shared rooms than in Open space.

Personally I prefer silence during work and have worked in startups with Cubicles, Office(great view of Mt. Rainier), Shared rooms & Open space and would rate individual offices as worse of the lot due to the walls, real and virtual.

How does one become good at problem solving?

Work at becoming a better solver in addition to solving new problems.
See Topcoder tutorials such as How to find solution and how to dissect a problem.

Use Charlie Munger‘s “Mental models” method as another approach, they are covered in this pdf http://www.focusinvestor.com/Foc… & better in his book, Poor Charlie’s Almanack, http://www.poorcharliesalmanack….

What tools should a modern Python developer have under his belt?

I would like to propose a 6-12 month list, instead of a 5 year list.

Find out & use
-details of python REPL including dir & help command

-the Python coding standards, PEP 8 & Python API documentation standard, PEP 257.

-Read pythonic code, options include Guido’s essays likehttp://www.python.org/doc/essays… or Dare Obasanjo’s python code and followup posts,

-Code to write: Runtime object alteration, even if you do not use it latter. Write an iterator.

-the Python standard library including string, date, persistence, regex, etc.

-create a python package & upload to pypi.

-write some unit tests with unittest\PyUnit\nosehttp://docs.python.org/library/u…

For using Python, consider Tornado (web framework) by the FriendFeed (nowFacebook Inc. (company)) team.

Go through their documentation & excellent demos, https://github.com/facebook/torn….
If you prefer a book, go through “Building the real time user experience” by Ted Roden

For using Ruby, you can use what VMware Cloud Foundry Platform-as-a-Service (PaaS) uses, Thin webserver, Ruby 1.9 fibers and async DB drivers, see this article http://www.igvita.com/2011/04/14…

For Ruby I have also seen references to orbited & juggernaut.

• • •
Next Page »
Powered by: WordPress Theme based on Sharepoint like theme from: ADMIN-BG