Life in Balance: Amit Chaudhary's Blog
• • • • •

Deep details of Python based Technology Stack at Votizen from the Inside Votizen post by Paul Stamatiou.

• Amazon Web Services
  • webservers – 2 large instances running Apache, Django, Python
  • load balancer – 1 large instance running HAProxy and Stunnel
  • asynchronous workers – 3 x-large instances running several hundred Celery workers each
  • reverse geocoding & districts – 3 x-large instances running what we call politicoder
  • elastic search for voters – 2 High-Memory Quadruple Extra Large instance running elastic search with entire corpus of voters loaded into memory (verify 50-75 people/second)
  • error reporting – 1 large instance running a sentry server
  • utility server – 1 large instance used for various opsy operations
  • db-misc server – 1 large instance running redis, memcached, and RabbitMQ
  • admin server – 1 large instance used for managing deployment and monitoring site, runs CruiseControl (for deployment) and Nagios (for monitoring)
• Puppet\Chef - We were previously using Puppet to manage server configurations in our admin repository, but are in the process of moving to Chef, as our new ops person prefers Chef. Both are Ruby-based tools, but are more mature than any comparable Python solution
• Python - We write almost all our code in Python 2.6, because Python gives you wings
• Django - This is one of the most fully-baked Python web frameworks and we have 2 core contributors on our team. Version 1.4 was just released, but we are still running on 1.3
• Redis - We use redis mostly to handle caching of large lists of ids, because it has really good set/list operations built into it.
• Celery - We have to do a log of user processing after they sign up and even when they sign in: fetch contacts on your social network, find your contacts districts and see if they are voters, find your districts and see if you are a voter
• RabbitMQ - The default message queue used by Celery, and also just a good queuing system. We only use it for celery.
• Memcached - Cache whatever you can, whenever you can. Django has good memcached support and we try to take advantage of it as much as possible.

We have unit testing and continuous integration, there’s no staging environment, everything goes straight to production. We have feature flagging now to limit development features to staff.

We’re looking to move to Vagrant so the development and production environments can be nearly identical.

Below are some Programming & Software Engineering Podcasts which I listen to during my commute

• Software Engineering Radio

This is one of my current favorites, topics tend to be general programming and at other times on Scalability, Architecture or New trends. Note: iTunes does not show all episodes due to an issue they are working on. Some episodes:

Episode 165: NoSQL and MongoDB with Dwight Merriman

Episode 162: Project Voldemort with Jay Kreps

Episode 160: AspectJ and Spring AOP with Ramnivas Laddad

• Pragmatic Podcasts
  

Covers a wide range of topics, many from their authors. Some episodes:

Jonathan Rasmusson Interview

Michael J Mangino on Facebook Applications

Michael Nygard Interview

• JavaWorld’s Java Technology Insider

Not updated for an year, but still good one, topics tend to be Java focused. Some episodes:

High scalability and Javawith Todd Hoff

Grails 1.1: A conversation with Graeme Rocher

• Java Posse

A Java focused podcast, more of panel discussion on current Java topics. Regularly updated.

• Stack OverFlow

The creators are from Windows background, the topics vary, the episodes are mix of discussion, news & interviews, and there are at times some good indepth interviews.

Podcast #59 had Damien Katz on CouchDB & Erlang.

More Podcasts:

• The Ruby on Rails Podcast. On Rails
  

• Herding Code. Windows Focus

• FLOSS. All about Free Libre Open Source Software

• Teach Me to Code (was RailsCoach)

A similar list, Accidental Technologist: My Updated Developer Podcast List.

Highlights from The Way I Work: Paul English, Co-founder Kayak

• Up everyday at 6:00, Email & then Yoga. Has a meditation room.

• We work really hard for 40 to 45 hours a week, but we believe in people having strong personal lives. Over the past six years, there have been maybe five times I’ve spoken with Steve before 8 a.m., after 5 p.m., or on the weekend.

• Always drives kid to school.

• We have offices in Connecticut, Massachusetts, and California. We started with the first two because my co-founder, Steve Hafner, lives in Connecticut, I live in Boston, and neither one of us wanted to move.

• We have an open office environment. our general philosophy is that an open environment facilitates intellectual intensity. Most engineers are introverted. Here, when people overhear a discussion, we encourage them to walk over and say, “There’s another way to do that.”

• The engineers and I handle customer support. If you make the engineers answer e-mails and phone calls from the customers, the second or third time they get the same question, they’ll actually stop what they’re doing and fix the code. Then we don’t have those questions anymore.

• Real time datamining and support information. We have four monitors in the office where you can see real-time streaming information about the site — how many visitors, how many click throughs. It also displays the last customer e-mail that came in and the photo of the employee who answered it. So you’re walking by and you see, “Oh, Dan just answered a question.” We developed our own customer support software. One of the things it does is randomly select an employee response to a customer and send that response out to the entire company and to all of our investors each day. It keeps us on our toes.

• I keep noon to 2 p.m. open, because I like going out to lunch. It’s also a time for me to socialize. We have a very active work force.

• I do all of the firing. At times, I’ve fired maybe one out of every three people I’ve hired. That might make people think I’m bad at hiring, but I think I’m quite good at hiring. The only way 100 people can ever build a larger company than one that has more than 8,000 people — that’s what Expedia has — is by hiring Olympic-quality, unbelievable all stars of technology.

• Every Tuesday night, I have an open dinner at my house. Anywhere between four and 15 of my relatives will show up for dinner. I’m not a great cook, but it’s fun to have people over.

• I read for an hour every night before going to bed. I love reading books by Indian authors. I’ll also read books about global health and Africa, as well as a murder mystery now and then. But I don’t like business books. There are so many things in life that are more interesting than business.

Toyota’s car plant in Fremont, CA (NUMMI) is closing or as per their website nummi.com, already closed.

What a shame. It had a great tour for everyone, where one saw the famed” Toyota Way(Link to Wikipedia)” at work including anyone stopping the assembly and a continuous line of trucks of South California every 30 minutes or so which included parts to be used in the next few hours, the JIT(Just In Time) assembly.

It had a right from Matrix kind of Robot section with machines welding and working on Car & Truck bodies at an amazing speed.

Here is a video from a very similar robot:

More information:

• Web Tour webpage from 2007 From Archive.org
• Edmunds.com story on the plant

Here is something I have used a few times. Yahoo Research has a TagMaps project, WorldExplorer which shows main tags for any place and pictures.
One can quickly decide what to visit in a place.

Here is an image which shows places in Santa Barbara

And here is an interactive Flash widget for Peru.

One of the researcher behind this is Mor Naaman

Try India and San Francisco, see if you can find Yoga in the Presidio or find out why Machu Pichcu does not show up in Peru.

The talks at Open Source Conference (OSCON 2009) by O’Reilly are 45 minute sessions. This year, the conference is in San Jose, CA from July 20 – 24, 2009.

I would like to see the following ones this year.

These ideas were triggered from my own search fora proposal and have some link to my own background of a Web services backend engineer on Linux.

On Grid Programming

• Real life MapReduce examples using Apache Hadoop with Java & Streaming API

This would cover pseudo and real code for actual real life examples for Map Reduce, going beyond what is there at the end of the Google Map Reduce white paper. These would be projects powered by Apache Hadoop.

• Tour of Facebook\Apache Thrift

This would cover different aspects including serialization across same language, data transfer using their transport API & across different languages

• HDFS setup and access from Java\Rails

This would be for those considering using it as a DFS without Map Reduce. Leo touched on the topic in the blog entry, Rearchitecting Twitter

General Programming

• Java web frameworks and how they fit in together?

Unlike say Rails, Java has a wide array of web frameworks, see Wikipedia’s list in Java enterprise platform category. It might make sense to highlight a few like Raible did earlier, Comparing JSF, Spring MVC, Stripes, Struts 2, Tapestry and Wicket, however showing different ones as used in the Terracota Exam App.

• Python libraries and packages you were not aware of

The title says it all.

• Becoming a better Python Programmer

Something beyond the coding standards and idioms in the PEPs.

Desktop & Tools

• Linux desktop applications you might not know about
  

Again the title says it all, for example, for me, this would cover Amarok for playing music files, TakeNote or Tuxcards for outline note taking, Freemind for mind mapping and more, this would begin where the Linux Journal Readers‘ Choice Awards 2008 end.

• Top Open Source Personal productivity tools

The title says it all

• Best Open Source Developer Tools

This is a tricky one and would require multiple programmers to come up with a complete list, a Kernel programmer would be happy using vi\emacs, while a Java one would use Eclipse or Netbeans and so on.

If you are interested in presenting these, do consider making a proposal at the OSCON 2009 Call for Participation. The last date is Feb 3rd 2009.

Presenters get a free pass to the regular conference.