April 16, 2012

Python based Technology Stack at Votizen

Category: Software development,Technology,Work — by Amit Chaudhary @ 6:52 pm

Deep details of Python based Technology Stack at Votizen from the Inside Votizen post by Paul Stamatiou.

  • Amazon Web Services
    • webservers – 2 large instances running Apache, Django, Python
    • load balancer – 1 large instance running HAProxy and Stunnel
    • asynchronous workers – 3 x-large instances running several hundred Celery workers each
    • reverse geocoding & districts – 3 x-large instances running what we call politicoder
    • elastic search for voters – 2 High-Memory Quadruple Extra Large instance running elastic search with entire corpus of voters loaded into memory (verify 50-75 people/second)
    • error reporting – 1 large instance running a sentry server
    • utility server – 1 large instance used for various opsy operations
    • db-misc server – 1 large instance running redis, memcached, and RabbitMQ
    • admin server – 1 large instance used for managing deployment and monitoring site, runs CruiseControl (for deployment) and Nagios (for monitoring)
  • Puppet\Chef - We were previously using Puppet to manage server configurations in our admin repository, but are in the process of moving to Chef, as our new ops person prefers Chef. Both are Ruby-based tools, but are more mature than any comparable Python solution
  • Python - We write almost all our code in Python 2.6, because Python gives you wings
  • Django - This is one of the most fully-baked Python web frameworks and we have 2 core contributors on our team. Version 1.4 was just released, but we are still running on 1.3
  • Redis - We use redis mostly to handle caching of large lists of ids, because it has really good set/list operations built into it.
  • Celery - We have to do a log of user processing after they sign up and even when they sign in: fetch contacts on your social network, find your contacts districts and see if they are voters, find your districts and see if you are a voter
  • RabbitMQ - The default message queue used by Celery, and also just a good queuing system. We only use it for celery.
  • Memcached - Cache whatever you can, whenever you can. Django has good memcached support and we try to take advantage of it as much as possible.

We have unit testing and continuous integration, there’s no staging environment, everything goes straight to production. We have feature flagging now to limit development features to staff.

We’re looking to move to Vagrant so the development and production environments can be nearly identical.

• • •

January 8, 2012

2012 Python Meme from Amit: HeapQ, Flask, Python on Quora & Python 3

Category: Software development — by Amit Chaudhary @ 4:18 pm
  1. What is the coolest Python application, framework, or library you
    have discovered in 2011?
  2. What new programming technique did you learn in 2011?
  3. What’s the name of the open source project you contributed the
    most in 2011? What did you do?
    • Nothing in particular
  4. What was the Python blog or website you read the most in 2011?
  5. What are the three top things you want to learn in 2012?
    • At the top is more public projects on github & else where.
    • Let doing (public projects) decide what I learn
    • SQLAlChemy
  6. What are the top software, app, or lib you wish someone would write in 2012?

Tarek started this series with these instructions:

  • copy-paste the questions and answer to them in your blog
  • tweet it with the #2012pythonmeme hashtag
• • •

December 22, 2011

Top TEDx Conferences courtesy Youtube API & Python

Category: Life,Personal development,Software development — by Amit Chaudhary @ 5:17 pm

For some good conferences, these were the conferences with most talks in the top 500 talks, the number of talks & video link to top talk from that conference.

TEDxTokyo,24

TEDxBuenosAires,18

TEDxCairo,17

TEDxParis,17

TEDxDubai,14

TEDxCaltech,12

TEDxNYED,10

TEDxEast,7

TEDxLahore,7

TEDxTaipei,7

I wrote couple of Python (programming language) scripts that used YouTube API. The approach, find TEDx conference with a larger number of high quality talks, the quality measured by view count.

Some additional notes:
-A check in top 200, brings in TEDxBerlin, TEDxCMU, TEDxNASA & TEDxToronto
-A check in top 1000 brings in TEDxSF, TEDxAmsterdam & TEDxAcademy
-The code is bunch of small Python scripts and is in github at https://github.com/amitch/ytube-tedx

• • •

August 11, 2011

On Quora: Workspace, Problem Solving, Tools for Python Developer, Real time in Python or Ruby

Category: Personal development,Software development,Work — by Amit Chaudhary @ 11:01 am
Rooms with different engineers followed by Open space are the best among the other choices. It
-Encourages collaborationidea flow & camaraderie.
-promotes visibility & transparency.
-is efficient in terms of cost & time to setup.

The biggest drawback is that it does not leave space (mental, sound & time kind of space) for flow to happen easily. Hence you will see productive programming start in late afternoon or lots of head phones. This is less so in shared rooms than in Open space.

Personally I prefer silence during work and have worked in startups with Cubicles, Office(great view of Mt. Rainier), Shared rooms & Open space and would rate individual offices as worse of the lot due to the walls, real and virtual.

How does one become good at problem solving?

Work at becoming a better solver in addition to solving new problems.
See Topcoder tutorials such as How to find solution and how to dissect a problem.
http://www.topcoder.com/tc?modul…

Use Charlie Munger‘s “Mental models” method as another approach, they are covered in this pdf http://www.focusinvestor.com/Foc… & better in his book, Poor Charlie’s Almanack, http://www.poorcharliesalmanack….

What tools should a modern Python developer have under his belt?

I would like to propose a 6-12 month list, instead of a 5 year list.

Find out & use
-details of python REPL including dir & help command

-the Python coding standards, PEP 8 & Python API documentation standard, PEP 257.

-Read pythonic code, options include Guido’s essays likehttp://www.python.org/doc/essays… or Dare Obasanjo’s python code and followup posts,
http://www.google.com/search?hl=…

-Code to write: Runtime object alteration, even if you do not use it latter. Write an iterator.

-the Python standard library including string, date, persistence, regex, etc.

-create a python package & upload to pypi.

-write some unit tests with unittest\PyUnit\nosehttp://docs.python.org/library/u…

For using Python, consider Tornado (web framework) by the FriendFeed (nowFacebook Inc. (company)) team.

Go through their documentation & excellent demos, https://github.com/facebook/torn….
If you prefer a book, go through “Building the real time user experience” by Ted Roden

For using Ruby, you can use what VMware Cloud Foundry Platform-as-a-Service (PaaS) uses, Thin webserver, Ruby 1.9 fibers and async DB drivers, see this article http://www.igvita.com/2011/04/14…

For Ruby I have also seen references to orbited & juggernaut.

• • •

July 28, 2010

Programming & Software Engineering Podcasts: SE Radio, Pragmatic, Java & more

Category: Internet,Software development,Technology — by Amit Chaudhary @ 9:21 pm

Below are some Programming & Software Engineering Podcasts which I listen to during my commute

This is one of my current favorites, topics tend to be general programming and at other times on Scalability, Architecture or New trends. Note: iTunes does not show all episodes due to an issue they are working on. Some episodes:

Episode 165: NoSQL and MongoDB with Dwight Merriman

Episode 162: Project Voldemort with Jay Kreps

Episode 160: AspectJ and Spring AOP with Ramnivas Laddad

Covers a wide range of topics, many from their authors. Some episodes:

Jonathan Rasmusson Interview

Michael J Mangino on Facebook Applications

Michael Nygard Interview

Not updated for an year, but still good one, topics tend to be Java focused. Some episodes:

High scalability and Javawith Todd Hoff

Grails 1.1: A conversation with Graeme Rocher

A Java focused podcast, more of panel discussion on current Java topics. Regularly updated.

The creators are from Windows background, the topics vary, the episodes are mix of discussion, news & interviews, and there are at times some good indepth interviews.

Podcast #59 had Damien Katz on CouchDB & Erlang.

More Podcasts:

  • FLOSS. All about Free Libre Open Source Software

A similar list, Accidental Technologist: My Updated Developer Podcast List.

• • •

March 20, 2010

Yoga & Ayurveda for Eyes incl. Allergies: Milk cotton balls, Berries & Daily Yoga

Category: Life,Software development,Spiritual,Work,Yoga — by Amit Chaudhary @ 9:23 pm

Bright Eyes by Catherine Guthrie, Yoga Journal

The top tips

  • Reduce burn, redness, irritation in eyes with milk-soaked cotton balls

If they burn or are bloodshot or light sensitive, an excess of pitta may be to blame. To counter it, lie down for 15 minutes with milk-soaked cotton balls on your closed lids. Cucumber slices will also do the trick.

My own experience, this works particularly well for me, making them feel much better. I have not tried the rest yet.

  • Vitamin C, vitamin E, and lutein as the best antioxidants for eye health

To infuse your diet with these nutrients, dish up spinach, broccoli, corn, strawberries and nuts. The researchers suggest at least 250 milligrams (mg) of vitamin C, 90 mg of vitamin E, and 3 mg of lutein daily.# These levels are higher than the government’s recommended dietary intake; hedge your bets with a daily multivitamin.

An hour of Yoga 5 times a week should help those sitting in front of monitors

Essentially, we become zombies in front of a glowing screen, blinking only three times a minute instead of the normal 20. The result? Dry eyes.

The class included asana, pranayama, and guided relaxation. Those in the other group spent equal time in the company’s recreation center talking to friends, working out, and watching TV. By study’s end, the yogis reported a 30 percent decline in eye problems like dry eye; eye complaints increased in the other group.

More tips which might work better for others

  • Refresh your eyes with water

No matter what your dominant dosha, you can refresh your eyes by splashing them gently with cool water, blinking seven times (once for each chakra, or energy center in the body), and rotating them in all directions.

  • Rose water for tired or crusty eyes

If you awaken to eyes that feel tired or more crusty than usual, a kapha imbalance may be to blame, says Malhotra, the author of Inner Beauty: Discover Natural Beauty and Well-Being with the Traditions of Ayurveda. To quell kapha, she suggests sprinkling the eyes with rose water. You can look for rose water in health food stores or Middle Eastern markets, or make your own by soaking an organically grown rose in filtered water overnight

  • Ghee (Clarified Butter) for Dry, itchy eyes

Dry, itchy eyes may signal that your vata is out of balance. To restore them, Malhotra recommends a home version of an Ayurvedic treatment called netra basti. To start, warm a quarter cup of ghee (clarified butter) over medium heat, cool it to room temperature, pour half the liquid into an eyecup (sold at drugstores), lean your head back, and bathe the eye for five to seven minutes. Repeat on the other eye using the remaining ghee. (This treatment can be messy, so do it in a bathroom, in clothes that can handle a few drops of ghee.)

What’s more, it’s a good idea to save this self-care routine until just before bedtime, because your vision will be clouded for a few minutes afterward.

• • •
Next Page »
Powered by: WordPress Theme based on Sharepoint like theme from: ADMIN-BG