Python Tips
Follow
Find
19.0K views | +0 today
Python Tips
Various news, tutorials, and other stuff about Python programming.
Curated by Mathieu D.
Your new post is loading...
Your new post is loading...
Scooped by Mathieu D.
Scoop.it!

JSON validation | Alex Conrad

I spend most of my time working with back-end services where JSON over HTTP is king. JSON in, JSON out. While processing requests, one of the steps is to validate them. The tool I use for validating JSON input is validictory. I am surprised how little known it is, yet so useful.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

OSDC Slides – High performance Python web apps

OSDC Slides – High performance Python web apps | Python Tips | Scoop.it

We gave a talk last month with Benoit in Paris about web servers.

 

The talk consisted of a first part were we’ve explained how a web server works, and we got into details like what’s a backlog etc. and explained how the technology evolved from a single CGI process to modern setups based on event loops or such things.

We’ve also presented the stack I’ve set up for the Firefox Sync server, and gave an overview about stress tests tools.

 

The bottom line is : use Nginx, Gunicorn and Gevent.

 

If you wonder why we have socks and shoes on our slides, it’s because “socket” sounds a lot like “chaussette” (means socks) in French, so it’s a (stupid but efficient) joke.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Alec Munro: Sorting out

As someone who almost exclusively uses high-level languages, the idea of writing my own sorting algorithm seems a bit silly. However, I can certainly appreciate the algorithms used for sorting, and it's a neat opportunity to try to document and implement these algorithms using methods I am familiar with.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Hooking up Whoosh and SQLalchemy (sawhoosh)

I was talking to Tim VanSteenburgh (we both do work for Geek.net) and he had a proof of concept for automatically updating a search index when a change happens to a model and pairing a unified search results page with that to make an easy one-stop search solution for a website. I thought this was a pretty cool idea, so I decided to do a implementation of it on top of the Pyramid framework and share it the readers of my blog. I choose Whoosh only so that I could have an easy way for users to try the project out. Since whoosh is pure python it installs when you run your setup.py develop. But this approach could be applied to any searching system. Tim was using solr.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Invent with Python: Recursion Explained with the Flood Fill Algorithm (and Zombies and Cats)

Invent with Python: Recursion Explained with the Flood Fill Algorithm (and Zombies and Cats) | Python Tips | Scoop.it

This is a programming tutorial for beginner and intermediate programmers who want to learn what recursion is. The programming language used for the examples is Python, but you can probably follow along if you know programming in some other language such as PHP or JavaScript. There’s a lot more information about recursion on the Wikipedia article: http://en.wikipedia.org/wiki/Recursion_(computer_science) But this guide is meant to be a more practical guide to show how handy recursion is.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Reloading Python Modules

Being able to reload code modules is one of the many nice features of Python. This allows developers to modify parts of a Python application while the interpreter is running. In general, all that needs to be done is pass a module object to the imp.reload() function (or just reload() in Python 2.x), and the module will be reloaded from its source file.

There are a few potential complications, however.

If any other code references symbols exported by the reloaded module, they may still be bound to the original code. For example, imagine if module A contains the constant INTERVAL = 5, and module B imports that constant into its namespace (from A import INTERVAL). If we change the constant to INTERVAL = 10 and just reload module A, any values in module B that were based on INTERVAL won't be updated to reflect its new value.

The solution to this problem is to also reload module B. But it's important to only reload module B after module A has been reloaded. Otherwise, it won't pick up the updated symbols.

PyUnit deals with a variation of this problem by introducing a rollback importer. That approach "rolls back" the set of imported modules to some previous state by overriding Python's global __import__ hook. PyUnit's solution is effective at restoring the interpreter's state to pre-test conditions, but it's not a general solution for live code reloading because the unloaded modules aren't automatically reloaded.

The following describes a general module reloading solution which aims to make the process automatic, transparent, and reliable.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Python Descriptors, Part 1 of 2

I rather enjoyed writing about a relatively underused feature yesterday, so today is more of the same. Of course, continuing with a focus on Django, today’s Python feature is also commonly used throughout a number of Django’s internals: descriptors. Python’s documentation on descriptors is rather sparse, though there’s a great writeup on it already. I won’t try to reinvent the wheel in its entirety here, I’ll just write up some basic details and how it can be used for Django.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Eli Bendersky: Length-prefix framing for protocol buffers

Eli Bendersky: Length-prefix framing for protocol buffers | Python Tips | Scoop.it
I personally find Google’s protocol buffers library (protobuf) extremely convenient for efficient serialization and de-serialization of structured data from multiple programming languages. protobufs are perfect for TCP/IP links in general and socket-based IPC in particular.

Framing (the method of dividing a long stream of bytes into discrete messages) isn’t immediately obvious with protobuf. What you get from a protobuf serialization is a binary buffer of data. You almost certainly want to send more than one such buffer over time, so how does your peer know when one message ends and another starts?
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Armin Ronacher: WSGI and the Pluggable Pipe Dream

As a Python web developer you are at one point confronted with the term “WSGI” (which after 6 years of existence still does not have an official truncation. Some rhyme it with “Whisky”, others just pronounce the abbreviation, other's just call it by the pep number 333). WSGI when it was created was a pretty awesome thing. It made it possible to use any Python web application with any webserver by specifying a gateway layer.

In fact, it became so popular and well supported in the Python world that similar protocols were created for other languages as well. One of the first was Rack for Ruby, then came Jack/JSGI for JavaScript, PSGI/Plack for Perl and many others. And yet ever since WSGI became from a niche protocol nobody knew to the protocol for web applications people tried to change and replace it. Why is that and why does nobody succeed in replacing it?
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

How to stress test your app using Funkload — part 1

How to stress test your app using Funkload — part 1 | Python Tips | Scoop.it
Funkload is a functional and stress test tool that can be used on your web applications.
It’s my favorite stress tool for these reasons:
stress tests are implemented as PyUnit tests, so they can also be used as functional teststhe test runner is very light, it’s dead easy to run a stress test on your box against a local appit’s simple to do a distributed test with a few extra options — Funkload will drive the other boxes using SSHFunkload provides nice reporting features out of the box: you can do trends and diff reportsYou can monitor the server being tested
This blog post demonstrates how to use Funkload on a dummy app and generate report. A second blog post will explain how to run the test using several test servers, and how to monitor the server that runs the application.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

A Python equivalent to PHP's openssl_open() function

A Python equivalent to PHP's openssl_open() function | Python Tips | Scoop.it
For the project I am currently working on, we had a need to decode data in Python that had been encoded using PHP’s openssl_seal() function. Python does not appear to have a direct equivalent function. The M2Crypto package can decode quite a few different formats, but openssl_seal() and openssl_open() aren’t included.
Fortunately, M2Crypto contains the components required to decode this data, but you have to jump through a few hoops to get there.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Guido van Rossum: Before Python

Guido van Rossum: Before Python | Python Tips | Scoop.it
This morning I had a chat with the students at Google's CAPE program. Since I wrote up what I wanted to say I figured I might as well blog it here. Warning: this is pretty unedited (or else it would never be published :-). I'm posting it in my "personal" blog instead of the "Python history" blog because it mostly touches on my career before Python. Here goes.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

PyCharm: New PyCharm 2.0 EAP build: Cython, CoffeeScript, code coverage

PyCharm: New PyCharm 2.0 EAP build: Cython, CoffeeScript, code coverage | Python Tips | Scoop.it

When we announced the previous EAP build, we mentioned that we had two new features with the letter ‘C’ in development. Now we’re happy to announce that we have not two, but three of them, and all of them are included in today’s EAP.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

This Python Has Venom! | Symantec Connect Community

This Python Has Venom! | Symantec Connect Community | Python Tips | Scoop.it

Here at Symantec Security Response we have discovered Python.Pytroj, a proof-of-concept virus that infects .pyc files with arbitrary code.

In case you are unfamiliar with Python here is a brief description. Python is a programming language which can run on most major operating systems. Python scripts have the extension .py and, once executed, create .pyc files—Python compiled files.

So a quick test to see how it works!.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Scalable Real-Time Web Architecture, Part 2: A Live Graph with Orbited, MorbidQ, and js.io

Scalable Real-Time Web Architecture, Part 2: A Live Graph with Orbited, MorbidQ, and js.io | Python Tips | Scoop.it

This tutorial will show you how to build a simple broadcast application for numerical data and represent that data graphically in the browser. The resulting user interface is very simplistic so that we can focus on the important parts of real-time applications. The methods from this tutorial could be extended to include any type of real-time data including monitoring, financial, or otherwise.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Django: Using Caching to Track Online Users

Recently I wanted a simple solution to track whether a user is online on a given Django site. The definition of "online" on a site is kind of ambiguous, so I'll define that a user is considered to be online if they have made any request to the site in the last five minutes.

 

I found that one approach is to use Django's caching framework to track when a user last accessed the site. For example, upon each request, I can have a middleware set the current time as a cache value associated with a given user. This allows us to store some basic information about logged-in user's online state without having to hit the database on each request and easily retrieve it by accessing the cache.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Create Facebook Application in Python using App Engine

Run with Friends is a sample Facebook Canvas Application written in Python running on Google App Engine. The application is designed to run within an iframe on a Facebook Canvas page.

 

Facebook provides a large array of integration points for applications and this sample application makes use of only a few of those capabilities. Some of the things that are covered here are:

* Web based Authentication, specifically using the JavaScript SDK and the signed_request.

* Making use of a Social Plugin, notably the Login with Faces plugin.

* Showing Facebook Dialogs to allow the user to post content back to Facebook.

* Making Graph API calls using OAuth 2.0.

* Making use of Real-time Updates in order to keep the cached data up-to-date.

 

You can try the sample at https://apps.facebook.com/runwithfriends/

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Dabeaz: An Inside Look at the GIL Removal Patch of Lore

As most Python programmers know, people love to hate the Global Interpreter Lock (GIL). Why can't it simply be removed? What's the problem? However, if you've been around the Python community long enough, you might also know that the GIL was already removed once before--specifically, by Greg Stein who created a patch against Python 1.4 in 1996. In fact, here's a link:

 

* http://www.python.org/ftp/python/contrib-09-Dec-1999/System/threading.tar.gz

 

This patch often gets mentioned in discussions regarding the GIL--especially as justification for keeping the GIL. For example, see the Python FAQ, the forum post It Isn't Easy to Remove the GIL or this mailing discussion on Free Threading. These discussions usually point out that the patch made the performance of single-threaded applications much worse--so much so that the patch couldn't be adopted. However, beyond that, technical details about the patch are somewhat sketchy.

 

Despite using Python since early 1996, I will freely admit that I never really knew the details of Greg's GIL removal patch. For the most part, I just vaguely knew that someone had attempted to remove the GIL, that it apparently killed the performance of single-threaded apps, and that it subsequently faded into oblivion. However, given my recent interest in making the GIL better, I thought it might be a fun challenge to turn on the time machine, see if I could actually find the patch, and compile a version of GIL-less Python in order to take a peek under the covers.

 

So, in this post, I'll do just that and try to offer some commentary on some of the more interesting and subtle aspects of the patch--in particular, aspects of it that seem especially tricky or problematic. Given the increased interest in concurrency, the GIL, and other matters, I hope that this information might be of some use to others, or at the very least, help explain why Python still has a GIL. Plus, as the saying goes, those who don't study the past are doomed to repeat it. So, let's jump in.

more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Python Descriptors, Part 2 of 2

Yesterday, I gave a basic overview of descriptors and what they can do, including a simple example to demonstrate one in action. That’s all well and good, but today I’ll explain how this can be genuinely useful in your apps, particularly when used in models.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Wayne Witzel: Pyramid and Traversal with a RESTful interface

When Pyramid was first being developed I was intrigued by the idea that I could create context aware views and use a host of methods to check permissions on those contexts, generate URLs based off those contexts, and auto-magically call the view required based on the context and the requested resource path.

So one of my first experiments with Pyramid was to implement proper resource urls for contexts in a RESTful fashion. Eventually I plan to do this for the entire collection as well, but for now all I need is the context level RESTful interface. The goal of which is to have URLs that go something like this.

/resource/id (GET) - default view of the resource
/resource/id/edit (GET) - the form that allows you to edit the resource
/resource/id/create (GET) - the form that allows you to edit the resource
/resource/id (PUT) - updates
/resource/id (POST) - create
/resource/id (DELETE) - delete

This ends up being pretty damn simple with Pyramid and Traversal and for those of you new to traversal or even those who aren't, I highly recommend reading the Much Ado About Traversal chapter in the Pyramid documentation. Also on a side note all of the snippets from this post are part of a real project called Stockpot and the code is freely available via SourceForge.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Eli Bendersky: Python unit testing: parametrized test cases

Python’s standard unittest library is great and I use it all the time. One thing missing from it, however, is a simple way of running parametrized test cases. In other words, you can’t easily pass arguments into a unittest.TestCase from outside.

Consider the use case: I have some TestCase I want to invoke several times, each time passing it a different argument.
One approach often mentioned is create a base TestCase for the bulk functionality and derive sub-classes from it for variations. But this isn’t flexible enough – what if you want to add parameters from the outside (command-line) or test with a large amount of parameters?

Fortunately, Python is dynamic enough (and unittest flexible enough) to allow a relatively straightforward solution.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Tarek Ziade: How to stress test your app using Funkload — part 2

Tarek Ziade: How to stress test your app using Funkload — part 2 | Python Tips | Scoop.it
If you want to hammer a server, you cannot run all the virtual concurrent users from a single box. You will end up eating all your resources and will not be able to simulate a very big load. If you need to run a couple of hundreds of concurrent users, one box is enough. But you’ll need more boxes to get more load, obviously. Depending on the complexity of the tests, I usually end up having at the most 200 CU (Concurrent Users) per node.
In the past, I was using Fabric to do distributed tests for Funkload, see here. But this is not needed anymore because Funkload has now a built-in support to distribute the charge amongst several nodes. It uses a similar technique that I used in Fabric, by driving the nodes via SSH using Paramiko. The benefit though, is that it does all the heavy lifting for you: your test suite gets deployed on the nodes, and the XML result files gets downloaded for you on the master.
All you have to do once the test is over, is to merge the XML files and generate reports. And the merging is taken care of by a Funkload script so…
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Gevent, Long-Polling & You - Toast Driven

Gevent, Long-Polling & You - Toast Driven | Python Tips | Scoop.it
A big focus lately (as far as web technology trends go) is the move to "real-time". "Real-time" is a loaded terms that can mean a lot of things to different people, but what is consistent is people's desire to always have the latest information, without a lot of effort on their part. Because, let's face it, funny cat pictures are srs bizness. Regardless of what data you want to push, the technique herein might be a good starting place.

This article is born out of the frustration of trying to find information on the Internet related to doing long-polling request with a simple WSGI backend. Many people have talked about this style of setup (such as Convore) but resources seem scarce. So this is the fruits of my trials to get a simple messaging daemon setup.
more...
No comment yet.
Scooped by Mathieu D.
Scoop.it!

Wayne Witzel: Pyramid and velruse for Google authentication

As I continue to work with Pyramid I find myself really enjoying the web development I am doing. Recently I needed to integrate Google OAuth (and eventually Facebook and Twitter) as part of the options for the user signup experience. I knew Chris Davies has done something similar recently based on some Google+ activity I had read of his, so while I was exploring options, I also spoke with him about his experience with python-openid and velruse. The feedback and examples he gave to me are pretty much the basis for what you are about to read on how to get Google OAuth working with your Pyramid application.
more...
No comment yet.