I Wear Pants
Now with Stuff and Things!
A blog by Peter Fein.
Follow @wearpants
The views expressed here do not represent my past, present or future employers, collectives, family, nation-state or houseplants. They are mine alone. Who's else would they be?
Twiggy Now Supports Python 2.6
July 12, 2011 at 06:40 PM | Tags: python, twiggy, loggingJust pushed out version 0.4.4 of Twiggy, a more Pythonic logger. This release adds support for Python 2.6. Get it from the usual place. Please let me know if you have any problems. I'll be giving an invited talk at the 2011 Scipy Conference as part of the Core Python track. Hope to see you there!
Note: for my readers more interested in activism than code, I'll be creating separate channels real soon now.
Sprint on Twiggy at Pycon
March 11, 2011 at 09:26 AM | Tags: python, twiggy, sprint, pycon2011, pyconThis post was import from an earlier version of this blog. Original here.
Just a heads up to anyone looking to sprint at Pycon - I'll be hacking with some others on Twiggy, a more Pythonic logger (see this introduction).This is great project for new sprinters. We'll be working on some new features:
- Thread/process context support
- New backends (mail, syslog, http)
- Stdlib logging compatability layer
- Perhaps port to Python 3
- Full bug list
Also note that project hosting for twiggy has moved from Google Code to Bitbucket: http://hg.wearpants.org/twiggy
Hope to see you!
Python Guru Seeks Fun, Profit
February 17, 2011 at 09:04 AM | Tags: resume, python, juju, twiggy, job, ctoThis post was import from an earlier version of this blog. Original here.
Just wanted to put the word out that I'm looking for work. I've been programming exclusively in Python for the past eight years, so if you need a veteran Python developer I'm your guy. My last gig was as the CTO of a search engine startup. I won't repeat my resume, but a few quick skills highlights:
- distributed computing: concurrency, queues, network programming, cloud
- NoSQL & SQL (both ORM and by hand)
- deep understanding of HTTP and REST
- machine learning and full text search, including Lucene/Solr
- systems architecture
I'm a regular speaker at Pycon & open source user groups. You might want to take a look at my code to get a sense of what I do.
I need to telecommute, as I'm stuck in rural Washington state through the summer. I'm open to a variety of arrangements, from short-term contracting to employment. I prefer smaller companies and am intrigued by the possibility of part-time work. All said, I'm pretty flexible, so don't hesitate to get in touch.
Nothing like a broken installer...
November 11, 2010 at 09:53 AM | Tags: packaging, python, sucks, twiggy, releaseThis post was import from an earlier version of this blog. Original here.
… to let you know you have users. ;–)
Please grab version 0.4.2 of Twiggy, the previous tarball was missing a bunch of files.
Python packaging sucks. Sucks. Really, let’s just throw in the towel and use CPAN.
Anyone written a package for testing that your packaging/release process went correctly? Seems crazy that we’d need something like that, but I’ve messed up releases so many times that a sanity checker would be nice.
These comments were imported from an earlier version of this blog.
Sucks? I'd be interested to hear the details.
I've worked on a couple of (really simple) Python packages (cobs, crcmod), and managed to get it down to a 1-step process for building each package, i.e.
setup.py sdist
setup.py bdist_msi
That doesn't suck at all, it's quite straight-forward. But then perhaps your package is more advanced than the one's I've worked with. Does your packaging process require a bunch of manual steps, and what stops them being automated? I'm interested to hear the details.
The trouble with saying “let's use CPAN” is that CPAN is code for several things at once.
* CPAN is a registry of packages. Many people think that's *all* CPAN is, and they say “oh Python already has PyPI, stop complaining”.
* CPAN is a *repository* of the friggin' *source code* to all those packages; if a package is in CPAN, you know exactly where to get the package and it can be installed. With PyPI, there's no guarantee the source code is anywhere.
* CPAN is a repository of *all free-software* packages; the vast majority are under GPL or Artistic License terms. The packages on PyPI could be under any license, and many of them are non-free.
* CPAN is a command-line tool, ‘cpan(1)’, for installing every CPAN package.
It's so seamless and, ironically, One Obvious Way To Do It, that people mean “the combination of CPAN and the ‘cpan(1)’ tool” when they say “CPAN”. That's all made possible because the above points are reliably true for CPAN. Because those are not true for PyPI, Python packaging sucks.
So what we actually need is something that is *all* of the above for Python, and is the expected, official, no-brainer location to publish and seek packages for Python. The trouble is, the people who might be interested in doing so rarely see the problem, because PyPI *appears* to solve the problem.
Sadly, I think the horse has bolted; the PyPI administrators actively resist efforts to change the policy to require source code for PyPI packages under free software licenses. It's just an link index, with optional file storage that may or may not be used.
I know exactly what you mean, Peter! The same thing happened to me with a package release on Saturday: some files were not included, so I remembered I had to add a MANIFEST.in, and, great, the files got included in my .tar.gz! Everything should be fine, right? Wrong. When the .tar.gz was actually installed, the files were missing — because I needed to add them as "package_data=[]" in addition to having them in MANIFEST.in.
When I release packages with the little "pyron" tool I made, there's no problem (it does both steps for you), but for bigger projects I like to use standard approaches — and therefore trip over the repetition that distutils requires.
By the way, under 2.7, Tarek has improved things so that things like package_data get added to the MANIFEST automatically — so maybe someday things will be easier.
@Craig:
The problem's not the setup.py commands, but writing the setup.py/MANIFEST.in. At a minimum, you've got version numbers in two places and list of files in two places, in addition to the files themselves. Totally violates DRY. Then you need to remember to upload your files & docs.
I swear, every time I sit down to write a setup.py for a new project it takes me at least half a day of re-reading the distutils docs and mucking with an edit-install-uninstall loop. And we haven't even gotten to setuptools. ;-(
@Ben:
I'm not really serious about using CPAN; it's more a straw man (though one that actually seems to walk around). That said, yeah, the CPAN model is nice b/c it's integrated and covers most of a distributor's needs in one place, including perldocs. CPAN also requires/validates PGP signatures, IIRC.
The problem with PyPI/setup.py is that it's only half of a release management tool. It lacks the hooks needed to support the other things you want to do when distributing - publish new docs, tag your VCS, upload your tarball (if you host elsewhere). Maybe I should write a Makefile to handle all this, but... ick.
@Brandon:
I actually think the implicit inclusion of files in your tarball that aren't in Manifest.in makes things worse - it leads to forgetting about the Manifest.in to begin with (which has it's own implicit includes). Someone really needs to `import this`.
"the PyPI administrators actively resist efforts to change the policy to require source code for PyPI packages under free software licenses".
What does the inclusion of non-free software have to do with PyPI's alleged suckage, let alone the OP's frustration ? Irrelevant.
The fetching-building-installing part of PyPI packages is almost a solved problem with pip/virtualenv (or buildout, no experience with it) and several mirrors. What's still a pain (and what the OP is complaining about) is the packaging part, how to wrap your code in a package to upload to PyPI (or your site) for others to install.
There's non-free software on PyPI? I've never noticed. I'll have to be more careful when I look for software there.
I've started adding Makefile rules for 'make distcheck' to verify that my MANIFEST.in is sane, because of the pain you're describing. You can find an example at http://bazaar.launchpad.net/~gtimelog-dev/gtimelog/trunk/annotate/head:/Makefile (Bazaar has horrible web URLs, I hope your URLizer won't choke on that colon in the middle). It's crazy, complicated and obfuscated, but at least it stops me from making broken releases on PyPI.
I haven't had a problem forgetting to include something in package_data yet. If I have, I'll start extending make distcheck to create a virtualenv, install my local sdist, and run the testsuite of the installed package.
Python packaging sucks, but at least it's getting better. Remember the time when we had no automated dependency resolution systems?
@Marius:
re: virtualenv + tests: yeah, I had thoughts along that direction. Though that only keeps your package from being totally borked, not from forgetting to include things that your tests don't capture (docs, random little scripts, whatever).
Really though, testing your package seems like trying to put gloves on after you've already got frostbite.
log.name("twiggy").info("What's new, what's next")
November 09, 2010 at 11:00 AM | Tags: python, release, logging, twiggyThis post was import from an earlier version of this blog. Original here.
An update about Twiggy, my new Pythonic logger.
What’s New
Yesterday I released a new version 0.4.1 of Twiggy. This release adds full test coverage (over 1000 lines, nearly twice the lines of actual code). I’ve fixed a number of important bugs in the process, so you’re encouraged to upgrade.
The features system is currently deprecated, pending a reimplementation in version 0.5. Features are currently global (shared by all log instances); they really should be per-object so libraries can use them without stepping on each other. Expect some clever metaprogramming voodoo to make this work while keeping things running fast.
What’s Next
Here’s a little preview of what you can expect over the next few weeks:
Be the best, steal from the rest
I’ll be adding support for context fields, a feature inspired by Logbook’s stacks. This allows an application to add fields to all log messages on a per-thread or per-process basis.
>>> from twiggy import * >>> quickSetup() >>> log.process(x=42) >>> log.thread(y=100) >>> log.debug('yo') DEBUG:x=42:y=100:yo >>> def doit(): ... log.debug('no y') ... log.thread(y=999) ... log.debug('different y') ... >>> import threading >>> t = threading.Thread(target=doit) >>> t.start(); t.join() DEBUG:x=42:no y DEBUG:x=42:y=999:different y
This is a killer feature for logging/debugging in webapps. One often wants to inject the request ID into all messages, including libraries that don’t know/care that they’re running on the web. There’ll be methods for clearing these contexts, as well as context managers to use with the with: statement.
Stdlib compatibility layer
0.5 will improve compatibility with the standard library’s logging package. This compatiblity will be two-way. You’ll be able to:
- configure twiggy to use stdlib logging as an output backend
- inject an API shim that emulates basic logging functionality
The later requires some explanation. 90-plus percent of the logging code I’ve ever seen only uses the most basic functionality: creating loggers, logging messages and capturing tracebacks. For such code, it should be possible to do:
from twiggy import logging_compat as logging log = logging.getLogger("oldcode") log.info("Shh, don't tell")
Even better, twiggy will provide a logging_compat.hijack() method to inject itself into sys.modules so that no modification to old code is needed at all.
I don’t expect this compatibility layer to work for everyone – notably, custom handlers won’t be supported (the underlying models are just too different), but this should ease the transition pain for many people.
Indentation
Also planned for 0.5 is support for user-defined counters. This feature is still taking shape, but it’ll look something like:
>>> def deep(): ... with log.increment('depth'): ... log.info("it's dark") ... abyss() ... log.warning("coming back up") ... >>> def abyss(): ... with log.increment('depth'): ... log.info("it's cold") ... >>> deep() INFO:depth=1:it's dark INFO:depth=2:it's cold WARNING:depth=1:coming back up
Outputs will be able to transform the depth field into useful visual formatting – for example, by using indentation to group lines together in a console app, or by setting a CSS class in HTML. Hell yeah, structured logging.
Etc.
Other forthcoming changes include: a port to Python 3, PEP-8 compliance, rewriting the features system, support for the warnings module and various minor enhancements. I’ll continue to support Python 2.7 using 3to2
N+1
I should probably stop there, but I’m excited by what’s further down the road. That includes:
- lazy logging: an output backend that groups messages together by a key, and only outputs them if some condition is met. For example, capture messages by request ID, and output all of them together if any one message is ERROR or higher.
- cluster logging: Twiggy will support easily settting up a master logging daemon to receive messages from multiple processes on a machine or across your cluster.
- unittest support: stuff the expected log output in your test docstring, apply a decorator, and Twiggy will add additional asserts to ensure your logs come out right.
- backends, backends, backends: email, HTTP, SQL, CouchDB, syslog, NT event log… Maybe even backends that open tickets in your bug tracker or stream live logs to your browser. Yeah.
What do you want?
Now is your opportunity to let me know what you want in a logger. Got a feature I haven’t thought of? Crazy idea? Think I should implement your favorite backend sooner? Tell me in the comments below.
Pete cooks, rides bikes and hacks Python. Maybe for you?. Don’t worry, he wears pants.
These comments were imported from an earlier version of this blog.
Phil Mayes 2011/02/17 12:46:35 -0800
Talk to Art and Logic, www.artlogic.com