Learning Python, 4th Edition

www.it-ebooks.infoLearning Python

www.it-ebooks.info

www.it-ebooks.info FOURTH EDITION Learning Python Mark LutzBeijing • Cambridge • Farnham • Köln • Sebastopol • Taipei • Tokyo

www.it-ebooks.infoLearning Python, Fourth Editionby Mark LutzCopyright © 2009 Mark Lutz. All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editionsare also available for most titles (http://my.safaribooksonline.com). For more information, contact ourcorporate/institutional sales department: (800) 998-9938 or [email protected]: Julie Steele Indexer: John BickelhauptProduction Editor: Sumita Mukherji Cover Designer: Karen MontgomeryCopyeditor: Rachel Head Interior Designer: David FutatoProduction Services: Newgen North America Illustrator: Robert RomanoPrinting History: First Edition. Second Edition. March 1999: Third Edition. December 2003: Fourth Edition. October 2007: September 2009:Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks ofO’Reilly Media, Inc. Learning Python, the image of a wood rat, and related trade dress are trademarksof O’Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of atrademark claim, the designations have been printed in caps or initial caps.While every precaution has been taken in the preparation of this book, the publisher and author assumeno responsibility for errors or omissions, or for damages resulting from the use of the information con-tained herein.ISBN: 978-0-596-15806-4[M]1252944666

www.it-ebooks.info To Vera.You are my life.

www.it-ebooks.info

www.it-ebooks.info Table of ContentsPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxiPart I. Getting Started1. A Python Q&A Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Why Do People Use Python? 3Software Quality 4Developer Productivity 5Is Python a “Scripting Language”? 5OK, but What’s the Downside? 7Who Uses Python Today? 7What Can I Do with Python? 9Systems Programming 9GUIs 9Internet Scripting 10Component Integration 10Database Programming 11Rapid Prototyping 11Numeric and Scientific Programming 11Gaming, Images, Serial Ports, XML, Robots, and More 12How Is Python Supported? 12What Are Python’s Technical Strengths? 13It’s Object-Oriented 13It’s Free 13It’s Portable 14It’s Powerful 15It’s Mixable 16It’s Easy to Use 16It’s Easy to Learn 17It’s Named After Monty Python 17How Does Python Stack Up to Language X? 17 vii

www.it-ebooks.infoChapter Summary 18Test Your Knowledge: Quiz 19Test Your Knowledge: Answers 192. How Python Runs Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Introducing the Python Interpreter 23Program Execution 24The Programmer’s View 24Python’s View 26Execution Model Variations 29Python Implementation Alternatives 29Execution Optimization Tools 30Frozen Binaries 32Other Execution Options 33Future Possibilities? 33Chapter Summary 34Test Your Knowledge: Quiz 34Test Your Knowledge: Answers 343. How You Run Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35The Interactive Prompt 35Running Code Interactively 37Why the Interactive Prompt? 38Using the Interactive Prompt 39System Command Lines and Files 41A First Script 42Running Files with Command Lines 43Using Command Lines and Files 44Unix Executable Scripts (#!) 46Clicking File Icons 47Clicking Icons on Windows 47The input Trick 49Other Icon-Click Limitations 50Module Imports and Reloads 51The Grander Module Story: Attributes 53import and reload Usage Notes 56Using exec to Run Module Files 57The IDLE User Interface 58IDLE Basics 58Using IDLE 60Advanced IDLE Tools 62Other IDEs 63Other Launch Options 64viii | Table of Contents

www.it-ebooks.info Embedding Calls 64 Frozen Binary Executables 65 Text Editor Launch Options 65 Still Other Launch Options 66 Future Possibilities? 66Which Option Should I Use? 66Chapter Summary 68Test Your Knowledge: Quiz 68Test Your Knowledge: Answers 69Test Your Knowledge: Part I Exercises 70Part II. Types and Operations4. Introducing Python Object Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75Why Use Built-in Types? 76Python’s Core Data Types 77Numbers 78Strings 80Sequence Operations 80Immutability 82Type-Specific Methods 82Getting Help 84Other Ways to Code Strings 85Pattern Matching 85Lists 86Sequence Operations 86Type-Specific Operations 87Bounds Checking 87Nesting 88Comprehensions 88Dictionaries 90Mapping Operations 90Nesting Revisited 91Sorting Keys: for Loops 93Iteration and Optimization 94Missing Keys: if Tests 95Tuples 96Why Tuples? 97Files 97Other File-Like Tools 99Other Core Types 99How to Break Your Code’s Flexibility 100 Table of Contents | ix

www.it-ebooks.info User-Defined Classes 101 And Everything Else 102Chapter Summary 103Test Your Knowledge: Quiz 103Test Your Knowledge: Answers 1045. Numeric Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105Numeric Type Basics 105Numeric Literals 106Built-in Numeric Tools 108Python Expression Operators 108Numbers in Action 113Variables and Basic Expressions 113Numeric Display Formats 115Comparisons: Normal and Chained 116Division: Classic, Floor, and True 117Integer Precision 121Complex Numbers 122Hexadecimal, Octal, and Binary Notation 122Bitwise Operations 124Other Built-in Numeric Tools 125Other Numeric Types 127Decimal Type 127Fraction Type 129Sets 133Booleans 139Numeric Extensions 140Chapter Summary 141Test Your Knowledge: Quiz 141Test Your Knowledge: Answers 1416. The Dynamic Typing Interlude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143The Case of the Missing Declaration Statements 143Variables, Objects, and References 144Types Live with Objects, Not Variables 145Objects Are Garbage-Collected 146Shared References 148Shared References and In-Place Changes 149Shared References and Equality 151Dynamic Typing Is Everywhere 152Chapter Summary 153Test Your Knowledge: Quiz 153Test Your Knowledge: Answers 154x | Table of Contents

www.it-ebooks.info7. Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155String Literals 157Single- and Double-Quoted Strings Are the Same 158Escape Sequences Represent Special Bytes 158Raw Strings Suppress Escapes 161Triple Quotes Code Multiline Block Strings 162Strings in Action 163Basic Operations 164Indexing and Slicing 165String Conversion Tools 169Changing Strings 171String Methods 172String Method Examples: Changing Strings 174String Method Examples: Parsing Text 176Other Common String Methods in Action 177The Original string Module (Gone in 3.0) 178String Formatting Expressions 179Advanced String Formatting Expressions 181Dictionary-Based String Formatting Expressions 182String Formatting Method Calls 183The Basics 184Adding Keys, Attributes, and Offsets 184Adding Specific Formatting 185Comparison to the % Formatting Expression 187Why the New Format Method? 190General Type Categories 193Types Share Operation Sets by Categories 194Mutable Types Can Be Changed In-Place 194Chapter Summary 195Test Your Knowledge: Quiz 195Test Your Knowledge: Answers 1968. Lists and Dictionaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197Lists 197Lists in Action 200Basic List Operations 200List Iteration and Comprehensions 200Indexing, Slicing, and Matrixes 201Changing Lists In-Place 202Dictionaries 207Dictionaries in Action 209Basic Dictionary Operations 209Changing Dictionaries In-Place 210 Table of Contents | xi

www.it-ebooks.info More Dictionary Methods 211 A Languages Table 212 Dictionary Usage Notes 213 Other Ways to Make Dictionaries 216 Dictionary Changes in Python 3.0 217Chapter Summary 223Test Your Knowledge: Quiz 224Test Your Knowledge: Answers 2249. Tuples, Files, and Everything Else . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225Tuples 225Tuples in Action 227Why Lists and Tuples? 229Files 229Opening Files 230Using Files 231Files in Action 232Other File Tools 238Type Categories Revisited 239Object Flexibility 241References Versus Copies 241Comparisons, Equality, and Truth 244Python 3.0 Dictionary Comparisons 246The Meaning of True and False in Python 246Python’s Type Hierarchies 248Type Objects 249Other Types in Python 250Built-in Type Gotchas 251Assignment Creates References, Not Copies 251Repetition Adds One Level Deep 252Beware of Cyclic Data Structures 252Immutable Types Can’t Be Changed In-Place 253Chapter Summary 253Test Your Knowledge: Quiz 254Test Your Knowledge: Answers 254Test Your Knowledge: Part II Exercises 255Part III. Statements and Syntax10. Introducing Python Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261Python Program Structure Revisited 261Python’s Statements 262xii | Table of Contents

www.it-ebooks.infoA Tale of Two ifs 264 What Python Adds 264 What Python Removes 265 Why Indentation Syntax? 266 A Few Special Cases 269 271A Quick Example: Interactive Loops 271 A Simple Interactive Loop 272 Doing Math on User Inputs 273 Handling Errors by Testing Inputs 274 Handling Errors with try Statements 275 Nesting Code Three Levels Deep 276 276Chapter Summary 277Test Your Knowledge: QuizTest Your Knowledge: Answers11. Assignments, Expressions, and Prints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279Assignment Statements 279Assignment Statement Forms 280Sequence Assignments 281Extended Sequence Unpacking in Python 3.0 284Multiple-Target Assignments 288Augmented Assignments 289Variable Name Rules 292Expression Statements 295Expression Statements and In-Place Changes 296Print Operations 297The Python 3.0 print Function 298The Python 2.6 print Statement 300Print Stream Redirection 302Version-Neutral Printing 306Chapter Summary 308Test Your Knowledge: Quiz 308Test Your Knowledge: Answers 30812. if Tests and Syntax Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311if Statements 311General Format 311Basic Examples 312Multiway Branching 312Python Syntax Rules 314Block Delimiters: Indentation Rules 315Statement Delimiters: Lines and Continuations 317A Few Special Cases 318 Table of Contents | xiii

www.it-ebooks.infoTruth Tests 320The if/else Ternary Expression 321Chapter Summary 324Test Your Knowledge: Quiz 324Test Your Knowledge: Answers 32413. while and for Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327while Loops 327General Format 328Examples 328break, continue, pass, and the Loop else 329General Loop Format 329pass 330continue 331break 331Loop else 332for Loops 334General Format 334Examples 335Loop Coding Techniques 341Counter Loops: while and range 342Nonexhaustive Traversals: range and Slices 343Changing Lists: range 344Parallel Traversals: zip and map 345Generating Both Offsets and Items: enumerate 348Chapter Summary 349Test Your Knowledge: Quiz 349Test Your Knowledge: Answers 35014. Iterations and Comprehensions, Part 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351Iterators: A First Look 351The Iteration Protocol: File Iterators 352Manual Iteration: iter and next 354Other Built-in Type Iterators 356List Comprehensions: A First Look 358List Comprehension Basics 359Using List Comprehensions on Files 359Extended List Comprehension Syntax 361Other Iteration Contexts 362New Iterables in Python 3.0 366The range Iterator 367The map, zip, and filter Iterators 368Multiple Versus Single Iterators 369xiv | Table of Contents

www.it-ebooks.info Dictionary View Iterators 370Other Iterator Topics 372Chapter Summary 372Test Your Knowledge: Quiz 372Test Your Knowledge: Answers 37315. The Documentation Interlude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375Python Documentation Sources 375# Comments 376The dir Function 376Docstrings: __doc__ 377PyDoc: The help Function 380PyDoc: HTML Reports 383The Standard Manual Set 386Web Resources 387Published Books 387Common Coding Gotchas 387Chapter Summary 389Test Your Knowledge: Quiz 389Test Your Knowledge: Answers 390Test Your Knowledge: Part III Exercises 390Part IV. Functions16. Function Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395Why Use Functions? 396Coding Functions 396def Statements 398def Executes at Runtime 399A First Example: Definitions and Calls 400Definition 400Calls 400Polymorphism in Python 401A Second Example: Intersecting Sequences 402Definition 402Calls 403Polymorphism Revisited 403Local Variables 404Chapter Summary 404Test Your Knowledge: Quiz 405Test Your Knowledge: Answers 405 Table of Contents | xv

www.it-ebooks.info17. Scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407Python Scope Basics 407Scope Rules 408Name Resolution: The LEGB Rule 410Scope Example 411The Built-in Scope 412The global Statement 414Minimize Global Variables 415Minimize Cross-File Changes 416Other Ways to Access Globals 418Scopes and Nested Functions 419Nested Scope Details 419Nested Scope Examples 419The nonlocal Statement 425nonlocal Basics 425nonlocal in Action 426Why nonlocal? 429Chapter Summary 432Test Your Knowledge: Quiz 433Test Your Knowledge: Answers 43418. Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435Argument-Passing Basics 435Arguments and Shared References 436Avoiding Mutable Argument Changes 438Simulating Output Parameters 439Special Argument-Matching Modes 440The Basics 441Matching Syntax 442The Gritty Details 443Keyword and Default Examples 444Arbitrary Arguments Examples 446Python 3.0 Keyword-Only Arguments 450The min Wakeup Call! 453Full Credit 454Bonus Points 455The Punch Line... 456Generalized Set Functions 456Emulating the Python 3.0 print Function 457Using Keyword-Only Arguments 459Chapter Summary 460Test Your Knowledge: Quiz 461Test Your Knowledge: Answers 462xvi | Table of Contents

www.it-ebooks.info19. Advanced Function Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 463Function Design Concepts 463Recursive Functions 465Summation with Recursion 465Coding Alternatives 466Loop Statements Versus Recursion 467Handling Arbitrary Structures 468Function Objects: Attributes and Annotations 469Indirect Function Calls 469Function Introspection 470Function Attributes 471Function Annotations in 3.0 472Anonymous Functions: lambda 474lambda Basics 474Why Use lambda? 475How (Not) to Obfuscate Your Python Code 477Nested lambdas and Scopes 478Mapping Functions over Sequences: map 479Functional Programming Tools: filter and reduce 481Chapter Summary 483Test Your Knowledge: Quiz 483Test Your Knowledge: Answers 48320. Iterations and Comprehensions, Part 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485List Comprehensions Revisited: Functional Tools 485List Comprehensions Versus map 486Adding Tests and Nested Loops: filter 487List Comprehensions and Matrixes 489Comprehending List Comprehensions 490Iterators Revisited: Generators 492Generator Functions: yield Versus return 492Generator Expressions: Iterators Meet Comprehensions 497Generator Functions Versus Generator Expressions 498Generators Are Single-Iterator Objects 499Emulating zip and map with Iteration Tools 500Value Generation in Built-in Types and Classes 5063.0 Comprehension Syntax Summary 507Comprehending Set and Dictionary Comprehensions 507Extended Comprehension Syntax for Sets and Dictionaries 508Timing Iteration Alternatives 509Timing Module 509Timing Script 510Timing Results 511 Table of Contents | xvii

www.it-ebooks.info Timing Module Alternatives 513 Other Suggestions 517Function Gotchas 518 Local Names Are Detected Statically 518 Defaults and Mutable Objects 520 Functions Without returns 522 Enclosing Scope Loop Variables 522Chapter Summary 522Test Your Knowledge: Quiz 523Test Your Knowledge: Answers 523Test Your Knowledge: Part IV Exercises 524Part V. Modules21. Modules: The Big Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529Why Use Modules? 529Python Program Architecture 530How to Structure a Program 531Imports and Attributes 531Standard Library Modules 533How Imports Work 5331. Find It 5342. Compile It (Maybe) 5343. Run It 535The Module Search Path 535Configuring the Search Path 537Search Path Variations 538The sys.path List 538Module File Selection 539Advanced Module Selection Concepts 540Chapter Summary 541Test Your Knowledge: Quiz 541Test Your Knowledge: Answers 54222. Module Coding Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543Module Creation 543Module Usage 544The import Statement 544The from Statement 545The from * Statement 545Imports Happen Only Once 546import and from Are Assignments 546xviii | Table of Contents

www.it-ebooks.info Cross-File Name Changes 547 import and from Equivalence 548 Potential Pitfalls of the from Statement 548Module Namespaces 550 Files Generate Namespaces 550 Attribute Name Qualification 552 Imports Versus Scopes 552 Namespace Nesting 553Reloading Modules 554 reload Basics 555 reload Example 556Chapter Summary 558Test Your Knowledge: Quiz 558Test Your Knowledge: Answers 55823. Module Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 561Package Import Basics 561Packages and Search Path Settings 562Package __init__.py Files 563Package Import Example 564from Versus import with Packages 566Why Use Package Imports? 566A Tale of Three Systems 567Package Relative Imports 569Changes in Python 3.0 570Relative Import Basics 570Why Relative Imports? 572The Scope of Relative Imports 574Module Lookup Rules Summary 575Relative Imports in Action 575Chapter Summary 581Test Your Knowledge: Quiz 582Test Your Knowledge: Answers 58224. Advanced Module Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 583Data Hiding in Modules 583Minimizing from * Damage: _X and __all__ 584Enabling Future Language Features 584Mixed Usage Modes: __name__ and __main__ 585Unit Tests with __name__ 586Using Command-Line Arguments with __name__ 587Changing the Module Search Path 590The as Extension for import and from 591 Table of Contents | xix

www.it-ebooks.infoModules Are Objects: Metaprograms 591Importing Modules by Name String 594Transitive Module Reloads 595Module Design Concepts 598Module Gotchas 599 599 Statement Order Matters in Top-Level Code 600 from Copies Names but Doesn’t Link 601 from * Can Obscure the Meaning of Variables 601 reload May Not Impact from Imports 602 reload, from, and Interactive Testing 603 Recursive from Imports May Not Work 604Chapter Summary 604Test Your Knowledge: Quiz 605Test Your Knowledge: Answers 605Test Your Knowledge: Part V ExercisesPart VI. Classes and OOP25. OOP: The Big Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 611Why Use Classes? 612OOP from 30,000 Feet 613Attribute Inheritance Search 613Classes and Instances 615Class Method Calls 616Coding Class Trees 616OOP Is About Code Reuse 619Chapter Summary 622Test Your Knowledge: Quiz 622Test Your Knowledge: Answers 62226. Class Coding Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 625Classes Generate Multiple Instance Objects 625Class Objects Provide Default Behavior 626Instance Objects Are Concrete Items 626A First Example 627Classes Are Customized by Inheritance 629A Second Example 630Classes Are Attributes in Modules 631Classes Can Intercept Python Operators 633A Third Example 634Why Use Operator Overloading? 636The World’s Simplest Python Class 636xx | Table of Contents

www.it-ebooks.info Classes Versus Dictionaries 639Chapter Summary 641Test Your Knowledge: Quiz 641Test Your Knowledge: Answers 64127. A More Realistic Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 643Step 1: Making Instances 644Coding Constructors 644Testing As You Go 645Using Code Two Ways 646Step 2: Adding Behavior Methods 648Coding Methods 649Step 3: Operator Overloading 651Providing Print Displays 652Step 4: Customizing Behavior by Subclassing 653Coding Subclasses 653Augmenting Methods: The Bad Way 654Augmenting Methods: The Good Way 654Polymorphism in Action 656Inherit, Customize, and Extend 657OOP: The Big Idea 658Step 5: Customizing Constructors, Too 658OOP Is Simpler Than You May Think 660Other Ways to Combine Classes 660Step 6: Using Introspection Tools 663Special Class Attributes 664A Generic Display Tool 665Instance Versus Class Attributes 666Name Considerations in Tool Classes 667Our Classes’ Final Form 668Step 7 (Final): Storing Objects in a Database 669Pickles and Shelves 670Storing Objects on a Shelve Database 671Exploring Shelves Interactively 672Updating Objects on a Shelve 674Future Directions 675Chapter Summary 677Test Your Knowledge: Quiz 677Test Your Knowledge: Answers 67828. Class Coding Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681The class Statement 681General Form 681 Table of Contents | xxi

www.it-ebooks.info Example 682Methods 684 685 Method Example 686 Calling Superclass Constructors 686 Other Method Call Possibilities 687Inheritance 687 Attribute Tree Construction 687 Specializing Inherited Methods 689 Class Interface Techniques 690 Abstract Superclasses 692 Python 2.6 and 3.0 Abstract Superclasses 693Namespaces: The Whole Story 693 Simple Names: Global Unless Assigned 693 Attribute Names: Object Namespaces 694 The “Zen” of Python Namespaces: Assignments Classify Names 696 Namespace Dictionaries 699 Namespace Links 701Documentation Strings Revisited 703Classes Versus Modules 703Chapter Summary 703Test Your Knowledge: Quiz 704Test Your Knowledge: Answers29. Operator Overloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 705The Basics 705Constructors and Expressions: __init__ and __sub__ 706Common Operator Overloading Methods 706Indexing and Slicing: __getitem__ and __setitem__ 708Intercepting Slices 708Index Iteration: __getitem__ 710Iterator Objects: __iter__ and __next__ 711User-Defined Iterators 712Multiple Iterators on One Object 714Membership: __contains__, __iter__, and __getitem__ 716Attribute Reference: __getattr__ and __setattr__ 718Other Attribute Management Tools 719Emulating Privacy for Instance Attributes: Part 1 720String Representation: __repr__ and __str__ 721Right-Side and In-Place Addition: __radd__ and __iadd__ 723In-Place Addition 725Call Expressions: __call__ 725Function Interfaces and Callback-Based Code 727Comparisons: __lt__, __gt__, and Others 728xxii | Table of Contents

www.it-ebooks.info The 2.6 __cmp__ Method (Removed in 3.0) 729Boolean Tests: __bool__ and __len__ 730Object Destruction: __del__ 732Chapter Summary 733Test Your Knowledge: Quiz 734Test Your Knowledge: Answers 73430. Designing with Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 737Python and OOP 737Overloading by Call Signatures (or Not) 738OOP and Inheritance: “Is-a” Relationships 739OOP and Composition: “Has-a” Relationships 740Stream Processors Revisited 742OOP and Delegation: “Wrapper” Objects 745Pseudoprivate Class Attributes 747Name Mangling Overview 748Why Use Pseudoprivate Attributes? 748Methods Are Objects: Bound or Unbound 750Unbound Methods are Functions in 3.0 752Bound Methods and Other Callable Objects 754Multiple Inheritance: “Mix-in” Classes 756Coding Mix-in Display Classes 757Classes Are Objects: Generic Object Factories 768Why Factories? 769Other Design-Related Topics 770Chapter Summary 770Test Your Knowledge: Quiz 770Test Your Knowledge: Answers 77131. Advanced Class Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773Extending Built-in Types 773Extending Types by Embedding 774Extending Types by Subclassing 775The “New-Style” Class Model 777New-Style Class Changes 778Type Model Changes 779Diamond Inheritance Change 783New-Style Class Extensions 788Instance Slots 788Class Properties 792__getattribute__ and Descriptors 794Metaclasses 794Static and Class Methods 795 Table of Contents | xxiii

www.it-ebooks.info Why the Special Methods? 795 Static Methods in 2.6 and 3.0 796 Static Method Alternatives 798 Using Static and Class Methods 799 Counting Instances with Static Methods 800 Counting Instances with Class Methods 802Decorators and Metaclasses: Part 1 804 Function Decorator Basics 804 A First Function Decorator Example 805 Class Decorators and Metaclasses 807 For More Details 808Class Gotchas 808 Changing Class Attributes Can Have Side Effects 808 Changing Mutable Class Attributes Can Have Side Effects, Too 810 Multiple Inheritance: Order Matters 811 Methods, Classes, and Nested Scopes 812 Delegation-Based Classes in 3.0: __getattr__ and built-ins 814 “Overwrapping-itis” 814Chapter Summary 815Test Your Knowledge: Quiz 815Test Your Knowledge: Answers 815Test Your Knowledge: Part VI Exercises 816Part VII. Exceptions and Tools32. Exception Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 825Why Use Exceptions? 825Exception Roles 826Exceptions: The Short Story 827Default Exception Handler 827Catching Exceptions 828Raising Exceptions 829User-Defined Exceptions 830Termination Actions 830Chapter Summary 833Test Your Knowledge: Quiz 833Test Your Knowledge: Answers 83333. Exception Coding Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 835The try/except/else Statement 835try Statement Clauses 837The try else Clause 839xxiv | Table of Contents

www.it-ebooks.info Example: Default Behavior 840 Example: Catching Built-in Exceptions 841The try/finally Statement 842 Example: Coding Termination Actions with try/finally 843Unified try/except/finally 844 Unified try Statement Syntax 845 Combining finally and except by Nesting 845 Unified try Example 846The raise Statement 848 Propagating Exceptions with raise 849 Python 3.0 Exception Chaining: raise from 849The assert Statement 850 Example: Trapping Constraints (but Not Errors!) 851with/as Context Managers 851 Basic Usage 852 The Context Management Protocol 853Chapter Summary 855Test Your Knowledge: Quiz 856Test Your Knowledge: Answers 85634. Exception Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 857Exceptions: Back to the Future 858String Exceptions Are Right Out! 858Class-Based Exceptions 859Coding Exceptions Classes 859Why Exception Hierarchies? 861Built-in Exception Classes 864Built-in Exception Categories 865Default Printing and State 866Custom Print Displays 867Custom Data and Behavior 868Providing Exception Details 868Providing Exception Methods 869Chapter Summary 870Test Your Knowledge: Quiz 871Test Your Knowledge: Answers 87135. Designing with Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 873Nesting Exception Handlers 873Example: Control-Flow Nesting 875Example: Syntactic Nesting 875Exception Idioms 877Exceptions Aren’t Always Errors 877 Table of Contents | xxv

www.it-ebooks.info Functions Can Signal Conditions with raise 878 Closing Files and Server Connections 878 Debugging with Outer try Statements 879 Running In-Process Tests 880 More on sys.exc_info 881Exception Design Tips and Gotchas 882 What Should Be Wrapped 882 Catching Too Much: Avoid Empty except and Exception 883 Catching Too Little: Use Class-Based Categories 885Core Language Summary 885 The Python Toolset 886 Development Tools for Larger Projects 887Chapter Summary 890Test Your Knowledge: Quiz 891Test Your Knowledge: Answers 891Test Your Knowledge: Part VII Exercises 891Part VIII. Advanced Topics36. Unicode and Byte Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 895String Changes in 3.0 896String Basics 897Character Encoding Schemes 897Python’s String Types 899Text and Binary Files 900Python 3.0 Strings in Action 902Literals and Basic Properties 902Conversions 903Coding Unicode Strings 904Coding ASCII Text 905Coding Non-ASCII Text 905Encoding and Decoding Non-ASCII text 906Other Unicode Coding Techniques 907Converting Encodings 909Coding Unicode Strings in Python 2.6 910Source File Character Set Encoding Declarations 912Using 3.0 Bytes Objects 913Method Calls 913Sequence Operations 914Other Ways to Make bytes Objects 915Mixing String Types 916Using 3.0 (and 2.6) bytearray Objects 917xxvi | Table of Contents

www.it-ebooks.infoUsing Text and Binary Files 920 Text File Basics 920 Text and Binary Modes in 3.0 921 Type and Content Mismatches 923 924Using Unicode Files 924 Reading and Writing Unicode in 3.0 926 Handling the BOM in 3.0 928 Unicode Files in 2.6 929 929Other String Tool Changes in 3.0 930 The re Pattern Matching Module 932 The struct Binary Data Module 934 The pickle Object Serialization Module 937 XML Parsing Tools 937 937Chapter SummaryTest Your Knowledge: QuizTest Your Knowledge: Answers37. Managed Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 941Why Manage Attributes? 941Inserting Code to Run on Attribute Access 942Properties 943The Basics 943A First Example 944Computed Attributes 945Coding Properties with Decorators 946Descriptors 947The Basics 948A First Example 950Computed Attributes 952Using State Information in Descriptors 953How Properties and Descriptors Relate 955__getattr__ and __getattribute__ 956The Basics 957A First Example 959Computed Attributes 961__getattr__ and __getattribute__ Compared 962Management Techniques Compared 963Intercepting Built-in Operation Attributes 966Delegation-Based Managers Revisited 970Example: Attribute Validations 973Using Properties to Validate 973Using Descriptors to Validate 975Using __getattr__ to Validate 977 Table of Contents | xxvii

www.it-ebooks.info Using __getattribute__ to Validate 978Chapter Summary 979Test Your Knowledge: Quiz 980 980 Test Your Knowledge: Answers38. Decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 983What’s a Decorator? 983Managing Calls and Instances 984Managing Functions and Classes 984Using and Defining Decorators 984Why Decorators? 985The Basics 986Function Decorators 986Class Decorators 990Decorator Nesting 993Decorator Arguments 994Decorators Manage Functions and Classes, Too 995Coding Function Decorators 996Tracing Calls 996State Information Retention Options 997Class Blunders I: Decorating Class Methods 1001Timing Calls 1006Adding Decorator Arguments 1008Coding Class Decorators 1011Singleton Classes 1011Tracing Object Interfaces 1013Class Blunders II: Retaining Multiple Instances 1016Decorators Versus Manager Functions 1018Why Decorators? (Revisited) 1019Managing Functions and Classes Directly 1021Example: “Private” and “Public” Attributes 1023Implementing Private Attributes 1023Implementation Details I 1025Generalizing for Public Declarations, Too 1026Implementation Details II 1029Open Issues 1030Python Isn’t About Control 1034Example: Validating Function Arguments 1034The Goal 1034A Basic Range-Testing Decorator for Positional Arguments 1035Generalizing for Keywords and Defaults, Too 1037Implementation Details 1040Open Issues 1042xxviii | Table of Contents

www.it-ebooks.info Decorator Arguments Versus Function Annotations 1043 Other Applications: Type Testing (If You Insist!) 1045Chapter Summary 1046Test Your Knowledge: Quiz 1047Test Your Knowledge: Answers 104739. Metaclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1051To Metaclass or Not to Metaclass 1052Increasing Levels of Magic 1052The Downside of “Helper” Functions 1054Metaclasses Versus Class Decorators: Round 1 1056The Metaclass Model 1058Classes Are Instances of type 1058Metaclasses Are Subclasses of Type 1061Class Statement Protocol 1061Declaring Metaclasses 1062Coding Metaclasses 1063A Basic Metaclass 1064Customizing Construction and Initialization 1065Other Metaclass Coding Techniques 1065Instances Versus Inheritance 1068Example: Adding Methods to Classes 1070Manual Augmentation 1070Metaclass-Based Augmentation 1071Metaclasses Versus Class Decorators: Round 2 1073Example: Applying Decorators to Methods 1076Tracing with Decoration Manually 1076Tracing with Metaclasses and Decorators 1077Applying Any Decorator to Methods 1079Metaclasses Versus Class Decorators: Round 3 1080Chapter Summary 1084Test Your Knowledge: Quiz 1084Test Your Knowledge: Answers 1085Part IX. Appendixes A. Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1089 B. Solutions to End-of-Part Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1101Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1139 Table of Contents | xxix

www.it-ebooks.info

www.it-ebooks.info PrefaceThis book provides an introduction to the Python programming language. Python is apopular open source programming language used for both standalone programs andscripting applications in a wide variety of domains. It is free, portable, powerful, andremarkably easy and fun to use. Programmers from every corner of the software in-dustry have found Python’s focus on developer productivity and software quality to bea strategic advantage in projects both large and small.Whether you are new to programming or are a professional developer, this book’s goalis to bring you quickly up to speed on the fundamentals of the core Python language.After reading this book, you will know enough about Python to apply it in whateverapplication domains you choose to explore.By design, this book is a tutorial that focuses on the core Python language itself, ratherthan specific applications of it. As such, it’s intended to serve as the first in a two-volumeset: • Learning Python, this book, teaches Python itself. • Programming Python, among others, shows what you can do with Python after you’ve learned it.That is, applications-focused books such as Programming Python pick up where thisbook leaves off, exploring Python’s role in common domains such as the Web, graphicaluser interfaces (GUIs), and databases. In addition, the book Python Pocket Referenceprovides additional reference materials not included here, and it is designed to sup-plement this book.Because of this book’s foundations focus, though, it is able to present Python funda-mentals with more depth than many programmers see when first learning the language.And because it’s based upon a three-day Python training class with quizzes and exer-cises throughout, this book serves as a self-paced introduction to the language. xxxi

www.it-ebooks.infoAbout This Fourth EditionThis fourth edition of this book has changed in three ways. This edition: • Covers both Python 3.0 and Python 2.6—it emphasizes 3.0, but notes differences in 2.6 • Includes a set of new chapters mainly targeted at advanced core-language topics • Reorganizes some existing material and expands it with new examples for clarityAs I write this edition in 2009, Python comes in two flavors—version 3.0 is an emergingand incompatible mutation of the language, and 2.6 retains backward compatibilitywith the vast body of existing Python code. Although Python 3 is viewed as the futureof Python, Python 2 is still widely used and will be supported in parallel with Python3 for years to come. While 3.0 is largely the same language, it runs almost no codewritten for prior releases (the mutation of print from statement to function alone,aesthetically sound as it may be, breaks nearly every Python program ever written).This split presents a bit of a dilemma for both programmers and book authors. Whileit would be easier for a book to pretend that Python 2 never existed and cover 3 only,this would not address the needs of the large Python user base that exists today. A vastamount of existing code was written for Python 2, and it won’t be going away any timesoon. And while newcomers to the language can focus on Python 3, anyone who mustuse code written in the past needs to keep one foot in the Python 2 world today. Sinceit may be years before all third-party libraries and extensions are ported to Python 3,this fork might not be entirely temporary.Coverage for Both 3.0 and 2.6To address this dichotomy and to meet the needs of all potential readers, this editionof this book has been updated to cover both Python 3.0 and Python 2.6 (and laterreleases in the 3.X and 2.X lines). It’s intended for programmers using Python 2, pro-grammers using Python 3, and programmers stuck somewhere between the two.That is, you can use this book to learn either Python line. Although the focus here ison 3.0 primarily, 2.6 differences and tools are also noted along the way for programmersusing older code. While the two versions are largely the same, they diverge in someimportant ways, and I’ll point these out along the way.For instance, I’ll use 3.0 print calls in most examples, but will describe the 2.6 printstatement, too, so you can make sense of earlier code. I’ll also freely introduce newfeatures, such as the nonlocal statement in 3.0 and the string format method in 2.6 and3.0, and will point out when such extensions are not present in older Pythons.If you are learning Python for the first time and don’t need to use any legacy code, Iencourage you to begin with Python 3.0; it cleans up some longstanding warts in thelanguage, while retaining all the original core ideas and adding some nice new tools.xxxii | Preface

www.it-ebooks.infoMany popular Python libraries and tools will likely be available for Python 3.0 by thetime you read these words, especially given the file I/O performance improvementsexpected in the upcoming 3.1 release. If you are using a system based on Python 2.X,however, you’ll find that this book addresses your concerns, too, and will help youmigrate to 3.0 in the future.By proxy, this edition addresses other Python version 2 and 3 releases as well, thoughsome older version 2.X code may not be able to run all the examples here. Althoughclass decorators are available in both Python 2.6 and 3.0, for example, you cannot usethem in an older Python 2.X that did not yet have this feature. See Tables P-1 and P-2later in this Preface for summaries of 2.6 and 3.0 changes. Shortly before going to press, this book was also augmented with notes about prominent extensions in the upcoming Python 3.1 release— comma separators and automatic field numbering in string format method calls, multiple context manager syntax in with statements, new methods for numbers, and so on. Because Python 3.1 was targeted pri- marily at optimization, this book applies directly to this new release as well. In fact, because Python 3.1 supersedes 3.0, and because the latest Python is usually the best Python to fetch and use anyhow, in this book the term “Python 3.0” generally refers to the language variations intro- duced by Python 3.0 but that are present in the entire 3.X line.New ChaptersAlthough the main purpose of this edition is to update the examples and material fromthe preceding edition for 3.0 and 2.6, I’ve also added five new chapters to address newtopics and add context: • Chapter 27 is a new class tutorial, using a more realistic example to explore the basics of Python object-oriented programming (OOP). • Chapter 36 provides details on Unicode and byte strings and outlines string and file differences between 3.0 and 2.6. • Chapter 37 collects managed attribute tools such as properties and provides new coverage of descriptors. • Chapter 38 presents function and class decorators and works through compre- hensive examples. • Chapter 39 covers metaclasses and compares and contrasts them with decorators.The first of these chapters provides a gradual, step-by-step tutorial for using classes andOOP in Python. It’s based upon a live demonstration I have been using in recent yearsin the training classes I teach, but has been honed here for use in a book. The chapteris designed to show OOP in a more realistic context than earlier examples and to Preface | xxxiii

www.it-ebooks.infoillustrate how class concepts come together into larger, working programs. I hope itworks as well here as it has in live classes.The last four of these new chapters are collected in a new final part of the book, “Ad-vanced Topics.” Although these are technically core language topics, not every Pythonprogrammer needs to delve into the details of Unicode text or metaclasses. Because ofthis, these four chapters have been separated out into this new part, and are officiallyoptional reading. The details of Unicode and binary data strings, for example, have beenmoved to this final part because most programmers use simple ASCII strings and don’tneed to know about these topics. Similarly, decorators and metaclasses are specialisttopics that are usually of more interest to API builders than application programmers.If you do use such tools, though, or use code that does, these new advanced topicchapters should help you master the basics. In addition, these chapters’ examples in-clude case studies that tie core language concepts together, and they are more sub-stantial than those in most of the rest of the book. Because this new part is optionalreading, it has end-of-chapter quizzes but no end-of-part exercises.Changes to Existing MaterialIn addition, some material from the prior edition has been reorganized, or supplemen-ted with new examples. Multiple inheritance, for instance, gets a new case study ex-ample that lists class trees in Chapter 30; new examples for generators that manuallyimplement map and zip are provided in Chapter 20; static and class methods are illus-trated by new code in Chapter 31; package relative imports are captured in action inChapter 23; and the __contains__, __bool__, and __index__ operator overloading meth-ods are illustrated by example now as well in Chapter 29, along with the newoverloading protocols for slicing and comparison.This edition also incorporates some reorganization for clarity. For instance, to accom-modate new material and topics, and to avoid chapter topic overload, five prior chaptershave been split into two each here. The result is new standalone chapters on operatoroverloading, scopes and arguments, exception statement details, and comprehensionand iteration topics. Some reordering has been done within the existing chapters aswell, to improve topic flow.This edition also tries to minimize forward references with some reordering, thoughPython 3.0’s changes make this impossible in some cases: to understand printing andthe string format method, you now must know keyword arguments for functions; tounderstand dictionary key lists and key tests, you must now know iteration; to useexec to run code, you need to be able to use file objects; and so on. A linear readingstill probably makes the most sense, but some topics may require nonlinear jumps andrandom lookups.All told, there have been hundreds of changes in this edition. The next section’s tablesalone document 27 additions and 57 changes in Python. In fact, it’s fair to say that thisxxxiv | Preface

www.it-ebooks.infoedition is somewhat more advanced, because Python is somewhat more advanced. Asfor Python 3.0 itself, though, you’re probably better off discovering most of this book’schanges for yourself, rather than reading about them further in this Preface.Specific Language Extensions in 2.6 and 3.0In general, Python 3.0 is a cleaner language, but it is also in some ways a more sophis-ticated language. In fact, some of its changes seem to assume you must already knowPython in order to learn Python! The prior section outlined some of the more prominentcircular knowledge dependencies in 3.0; as a random example, the rationale for wrap-ping dictionary views in a list call is incredibly subtle and requires substantial fore-knowledge. Besides teaching Python fundamentals, this book serves to help bridge thisknowledge gap.Table P-1 lists the most prominent new language features covered in this edition, alongwith the primary chapters in which they appear.Table P-1. Extensions in Python 2.6 and 3.0 Covered in chapter(s) 11 Extension 17 The print function in 3.0 7 The nonlocal x,y statement in 3.0 7, 36 The str.format method in 2.6 and 3.0 9, 36 String types in 3.0: str for Unicode text, bytes for binary data 31, 38 Text and binary file distinctions in 3.0 14, 20 Class decorators in 2.6 and 3.0: @private('age') 8, 14 New iterators in 3.0: range, map, zip 5 Dictionary views in 3.0: D.keys, D.values, D.items 5 Division operators in 3.0: remainders, / and // 4, 5, 14, 20 Set literals in 3.0: {a, b, c} 4, 8, 14, 20 Set comprehensions in 3.0: {x**2 for x in seq} 5 Dictionary comprehensions in 3.0: {x: x**2 for x in seq} 5 Binary digit-string support in 2.6 and 3.0: 0b0101, bin(I) 19 The fraction number type in 2.6 and 3.0: Fraction(1, 3) 18, 20 Function annotations in 3.0: def f(a:99, b:str)->int 11, 13 Keyword-only arguments in 3.0: def f(a, *b, c, **d) 23 Extended sequence unpacking in 3.0: a, *b = seq 33, 35 Relative import syntax for packages enabled in 3.0: from . 33, 34 Context managers enabled in 2.6 and 3.0: with/as Exception syntax changes in 3.0: raise, except/as, superclass Preface | xxxv

www.it-ebooks.infoExtension Covered in chapter(s)Exception chaining in 3.0: raise e2 from e1 33Reserved word changes in 2.6 and 3.0 11New-style class cutover in 3.0 31Property decorators in 2.6 and 3.0: @property 37Descriptor use in 2.6 and 3.0 31, 38Metaclass use in 2.6 and 3.0 31, 39Abstract base classes support in 2.6 and 3.0 28Specific Language Removals in 3.0In addition to extensions, a number of language tools have been removed in 3.0 in aneffort to clean up its design. Table P-2 summarizes the changes that impact this book,covered in various chapters of this edition. Many of the removals listed in Table P-2have direct replacements, some of which are also available in 2.6 to support futuremigration to 3.0.Table P-2. Removals in Python 3.0 that impact this bookRemoved Replacement Covered in chapter(s)reload(M) imp.reload(M) (or exec) 3, 22apply(f, ps, ks) f(*ps, **ks) 18 5`X` repr(X) 5 5X <> Y X != Y 5 8long int 3, 10 39999L 9999 14D.has_key(K) K in D (or D.get(key) != None) 9 14, 20, 29raw_input input 7, 29old input eval(input()) 7, 29 14, 19xrange range 3file open (and io module classes) 3X.next X.__next__, called by next(X) 5X.__getslice__ X.__getitem__ passed a slice object 11X.__setslice__ X.__setitem__ passed a slice objectreduce functools.reduce (or loop code)execfile(filename) exec(open(filename).read())exec open(filename) exec(open(filename).read())0777 0o777print x, y print(x, y)xxxvi | Preface

www.it-ebooks.infoRemoved Replacement Covered in chapter(s)print >> F, x, y print(x, y, file=F) 11print x, y, print(x, y, end=' ') 11u'ccc' 'ccc' 7, 36'bbb' for byte strings b'bbb' 7, 9, 36raise E, V raise E(V) 32, 33, 34except E, X: except E as X: 32, 33, 34def f((a, b)): def f(x): (a, b) = x 11, 18, 20file.xreadlines for line in file: (or X=iter(file)) 13, 14D.keys(), etc. as lists list(D.keys()) (dictionary views) 8, 14map(), range(), etc. as lists list(map()), list(range()) (built-ins) 14map(None, ...) zip (or manual code to pad results) 13, 20X=D.keys(); X.sort() sorted(D) (or list(D.keys())) 4, 8, 14cmp(x, y) (x > y) - (x < y) 29X.__cmp__(y) __lt__, __gt__, __eq__, etc. 29X.__nonzero__ X.__bool__ 29X.__hex__, X.__oct__ X._index__ 29Sort comparison functions Use key=transform or reverse=True 8Dictionary <, >, <=, >= Compare sorted(D.items()) (or loop code) 8, 9types.ListType list (types is for nonbuilt-in names only) 9__metaclass__ = M class C(metaclass=M): 28, 31, 39__builtin__ builtins (renamed) 17Tkinter tkinter (renamed) 18, 19, 24, 29, 30sys.exc_type, exc_value sys.exc_info()[0], [1] 34, 35function.func_code function.__code__ 19, 38__getattr__ run by built-ins Redefine __X__ methods in wrapper classes 30, 37, 38-t, –tt command-line switches Inconsistent tabs/spaces use is always an error 10, 12from ... *, within a function May only appear at the top level of a file 22import mod, in same package from . import mod, package-relative form 23class MyException: class MyException(Exception): 34exceptions module Built-in scope, library manual 34thread, Queue modules _thread, queue (both renamed) 17anydbm module dbm (renamed) 27cPickle module _pickle (renamed, used automatically) 9os.popen2/3/4 subprocess.Popen (os.popen retained) 14String-based exceptions Class-based exceptions (also required in 2.6) 32, 33, 34 Preface | xxxvii

www.it-ebooks.infoRemoved Replacement Covered in chapter(s)String module functions String object methods 7Unbound methods Functions (staticmethod to call via instance) 30, 31Mixed type comparisons, sorts Nonnumeric mixed type comparisons are errors 5, 9There are additional changes in Python 3.0 that are not listed in this table, simplybecause they don’t affect this book. Changes in the standard library, for instance, mighthave a larger impact on applications-focused books like Programming Python than theydo here; although most standard library functionality is still present, Python 3.0 takesfurther liberties with renaming modules, grouping them into packages, and so on. Fora more comprehensive list of changes in 3.0, see the “What’s New in Python 3.0”document in Python’s standard manual set.If you are migrating from Python 2.X to Python 3.X, be sure to also see the 2to3 auto-matic code conversion script that is available with Python 3.0. It can’t translate every-thing, but it does a reasonable job of converting the majority of 2.X code to run under3.X. As I write this, a new 3to2 back-conversion project is also underway to translatePython 3.X code to run in 2.X environments. Either tool may prove useful if you mustmaintain code for both Python lines; see the Web for details.Because this fourth edition is mostly a fairly straightforward update for 3.0 with ahandful of new chapters, and because it’s only been two years since the prior editionwas published, the rest of this Preface is taken from the prior edition with only minorupdating.About The Third EditionIn the four years between the publication of the second and third editions of this bookthere were substantial changes in Python itself, and in the topics I presented in Pythontraining sessions. The third edition reflected these changes, and also incorporated ahandful of structural changes.The Third Edition’s Python Language ChangesOn the language front, the third edition was thoroughly updated to reflect Python 2.5and all changes to the language since the publication of the second edition in late 2003.(The second edition was based largely on Python 2.2, with some 2.3 features graftedon at the end of the project.) In addition, discussions of anticipated changes in theupcoming Python 3.0 release were incorporated where appropriate. Here are some ofthe major language topics for which new or expanded coverage was provided (chapternumbers here have been updated to reflect the fourth edition):xxxviii | Preface

www.it-ebooks.info • The new B if A else C conditional expression (Chapter 19) • with/as context managers (Chapter 33) • try/except/finally unification (Chapter 33) • Relative import syntax (Chapter 23) • Generator expressions (Chapter 20) • New generator function features (Chapter 20) • Function decorators (Chapter 31) • The set object type (Chapter 5) • New built-in functions: sorted, sum, any, all, enumerate (Chapters 13 and 14) • The decimal fixed-precision object type (Chapter 5) • Files, list comprehensions, and iterators (Chapters 14 and 20) • New development tools: Eclipse, distutils, unittest and doctest, IDLE enhance- ments, Shedskin, and so on (Chapters 2 and 35)Smaller language changes (for instance, the widespread use of True and False; the newsys.exc_info for fetching exception details; and the demise of string-based exceptions,string methods, and the apply and reduce built-ins) are discussed throughout the book.The third edition also expanded coverage of some of the features that were new in thesecond edition, including three-limit slices and the arbitrary arguments call syntax thatsubsumed apply.The Third Edition’s Python Training ChangesBesides such language changes, the third edition was augmented with new topics andexamples presented in my Python training sessions. Changes included (chapter num-bers again updated to reflect those in the fourth edition): • A new chapter introducing built-in types (Chapter 4) • A new chapter introducing statement syntax (Chapter 10) • A new full chapter on dynamic typing, with enhanced coverage (Chapter 6) • An expanded OOP introduction (Chapter 25) • New examples for files, scopes, statement nesting, classes, exceptions, and moreMany additions and changes were made with Python beginners in mind, and sometopics were moved to appear at the places where they proved simplest to digest intraining classes. List comprehensions and iterators, for example, now make their initialappearance in conjunction with the for loop statement, instead of later with functionaltools. Preface | xxxix

www.it-ebooks.infoCoverage of many original core language topics also was substantially expanded in thethird edition, with new discussions and examples added. Because this text has becomesomething of a de facto standard resource for learning the core Python language, thepresentation was made more complete and augmented with new use cases throughout.In addition, a new set of Python tips and tricks, gleaned from 10 years of teaching classesand 15 years of using Python for real work, was incorporated, and the exercises wereupdated and expanded to reflect current Python best practices, new language features,and common beginners’ mistakes witnessed firsthand in classes. Overall, the core lan-guage coverage was expanded.The Third Edition’s Structural ChangesBecause the material was more complete, it was split into bite-sized chunks. The corelanguage material was organized into many multichapter parts to make it easier totackle. Types and statements, for instance, are now two top-level parts, with one chap-ter for each major type and statement topic. Exercises and “gotchas” (common mis-takes) were also moved from chapter ends to part ends, appearing at the end of the lastchapter in each part.In the third edition, I also augmented the end-of-part exercises with end-of-chaptersummaries and end-of-chapter quizzes to help you review chapters as you completethem. Each chapter concludes with a set of questions to help you review and test yourunderstanding of the chapter’s material. Unlike the end-of-part exercises, whose solu-tions are presented in Appendix B, the solutions to the end-of-chapter quizzes appearimmediately after the questions; I encourage you to look at the solutions even if you’resure you’ve answered the questions correctly because the answers are a sort of reviewin themselves.Despite all the new topics, the book is still oriented toward Python newcomers and isdesigned to be a first Python text for programmers. Because it is largely based on time-tested training experience and materials, it can still serve as a self-paced introductoryPython class.The Third Edition’s Scope ChangesAs of its third edition, this book is intended as a tutorial on the core Python language,and nothing else. It’s about learning the language in an in-depth fashion, before ap-plying it in application-level programming. The presentation here is bottom-up andgradual, but it provides a complete look at the entire language, in isolation from itsapplication roles.For some, “learning Python” involves spending an hour or two going through a tutorialon the Web. This works for already advanced programmers, up to a point; Python is,after all, relatively simple in comparison to other languages. The problem with this fast-track approach is that its practitioners eventually stumble onto unusual cases and getxl | Preface

www.it-ebooks.infostuck—variables change out from under them, mutable default arguments mutate in-explicably, and so on. The goal here is instead to provide a solid grounding in Pythonfundamentals, so that even the unusual cases will make sense when they crop up.This scope is deliberate. By restricting our gaze to language fundamentals, we can in-vestigate them here in more satisfying depth. Other texts, described ahead, pick upwhere this book leaves off and provide a more complete look at application-level topicsand additional reference materials. The purpose of the book you are reading now issolely to teach Python itself so that you can apply it to whatever domain you happento work in.About This BookThis section underscores some important points about this book in general, regardlessof its edition number. No book addresses every possible audience, so it’s important tounderstand a book’s goals up front.This Book’s PrerequisitesThere are no absolute prerequisites to speak of, really. Both true beginners and crustyprogramming veterans have used this book successfully. If you are motivated to learnPython, this text will probably work for you. In general, though, I have found that anyexposure to programming or scripting before this book can be helpful, even if notrequired for every reader.This book is designed to be an introductory-level Python text for programmers.* It maynot be an ideal text for someone who has never touched a computer before (for instance,we’re not going to spend any time exploring what a computer is), but I haven’t mademany assumptions about your programming background or education.On the other hand, I won’t insult readers by assuming they are “dummies,” either,whatever that means—it’s easy to do useful things in Python, and this book will showyou how. The text occasionally contrasts Python with languages such as C, C++, Java,and Pascal, but you can safely ignore these comparisons if you haven’t used such lan-guages in the past.This Book’s Scope and Other BooksAlthough this book covers all the essentials of the Python language, I’ve kept its scopenarrow in the interests of speed and size. To keep things simple, this book focuses oncore concepts, uses small and self-contained examples to illustrate points, and* And by “programmers,” I mean anyone who has written a single line of code in any programming or scripting language in the past. If this doesn’t include you, you will probably find this book useful anyhow, but be aware that it will spend more time teaching Python than programming fundamentals. Preface | xli

www.it-ebooks.infosometimes omits the small details that are readily available in reference manuals. Be-cause of that, this book is probably best described as an introduction and a stepping-stone to more advanced and complete texts.For example, we won’t talk much about Python/C integration—a complex topic thatis nevertheless central to many Python-based systems. We also won’t talk much aboutPython’s history or development processes. And popular Python applications such asGUIs, system tools, and network scripting get only a short glance, if they are mentionedat all. Naturally, this scope misses some of the big picture.By and large, Python is about raising the quality bar a few notches in the scripting world.Some of its ideas require more context than can be provided here, and I’d be remiss ifI didn’t recommend further study after you finish this book. I hope that most readersof this book will eventually go on to gain a more complete understanding of application-level programming from other texts.Because of its beginner’s focus, Learning Python is designed to be naturally comple-mented by O’Reilly’s other Python books. For instance, Programming Python, anotherbook I authored, provides larger and more complete examples, along with tutorials onapplication programming techniques, and was explicitly designed to be a follow-uptext to the one you are reading now. Roughly, the current editions of LearningPython and Programming Python reflect the two halves of their author’s trainingmaterials—the core language, and application programming. In addition, O’Reilly’sPython Pocket Reference serves as a quick reference supplement for looking up someof the finer details skipped here.Other follow-up books can also provide references, additional examples, or detailsabout using Python in specific domains such as the Web and GUIs. For instance,O’Reilly’s Python in a Nutshell and Sams’s Python Essential Reference serve as usefulreferences, and O’Reilly’s Python Cookbook offers a library of self-contained examplesfor people already familiar with application programming techniques. Because readingbooks is such a subjective experience, I encourage you to browse on your own to findadvanced texts that suit your needs. Regardless of which books you choose, though,keep in mind that the rest of the Python story requires studying examples that are morerealistic than there is space for here.Having said that, I think you’ll find this book to be a good first text on Python, despiteits limited scope (and perhaps because of it). You’ll learn everything you need to getstarted writing useful standalone Python programs and scripts. By the time you’ve fin-ished this book, you will have learned not only the language itself, but also how to applyit well to your day-to-day tasks. And you’ll be equipped to tackle more advanced topicsand examples as they come your way.xlii | Preface

www.it-ebooks.infoThis Book’s Style and StructureThis book is based on training materials developed for a three-day hands-on Pythoncourse. You’ll find quizzes at the end of each chapter, and exercises at the end of thelast chapter of each part. Solutions to chapter quizzes appear in the chapters themselves,and solutions to part exercises show up in Appendix B. The quizzes are designed toreview material, while the exercises are designed to get you coding right away and areusually one of the highlights of the course.I strongly recommend working through the quizzes and exercises along the way, notonly to gain Python programming experience, but also because some of the exercisesraise issues not covered elsewhere in the book. The solutions in the chapters and inAppendix B should help you if you get stuck (and you are encouraged to peek at theanswers as much and as often as you like).The overall structure of this book is also derived from class materials. Because this textis designed to introduce language basics quickly, I’ve organized the presentation bymajor language features, not examples. We’ll take a bottom-up approach here: frombuilt-in object types, to statements, to program units, and so on. Each chapter is fairlyself-contained, but later chapters draw upon ideas introduced in earlier ones (e.g., bythe time we get to classes, I’ll assume you know how to write functions), so a linearreading makes the most sense for most readers.In general terms, this book presents the Python language in a linear fashion. It is or-ganized with one part per major language feature—types, functions, and so forth—andmost of the examples are small and self-contained (some might also call the examplesin this text artificial, but they illustrate the points it aims to make). More specifically,here is what you will find:Part I, Getting Started We begin with a general overview of Python that answers commonly asked initial questions—why people use the language, what it’s useful for, and so on. The first chapter introduces the major ideas underlying the technology to give you some background context. Then the technical material of the book begins, as we explore the ways that both we and Python run programs. The goal of this part of the book is to give you just enough information to be able to follow along with later examples and exercises.Part II, Types and Operations Next, we begin our tour of the Python language, studying Python’s major built-in object types in depth: numbers, lists, dictionaries, and so on. You can get a lot done in Python with these tools alone. This is the most substantial part of the book because we lay groundwork here for later chapters. We’ll also look at dynamic typing and its references—keys to using Python well—in this part. Preface | xliii

www.it-ebooks.infoPart III, Statements and Syntax The next part moves on to introduce Python’s statements—the code you type to create and process objects in Python. It also presents Python’s general syntax model. Although this part focuses on syntax, it also introduces some related tools, such as the PyDoc system, and explores coding alternatives.Part IV, Functions This part begins our look at Python’s higher-level program structure tools. Func- tions turn out to be a simple way to package code for reuse and avoid code redun- dancy. In this part, we will explore Python’s scoping rules, argument-passing techniques, and more.Part V, Modules Python modules let you organize statements and functions into larger components, and this part illustrates how to create, use, and reload modules. We’ll also look at some more advanced topics here, such as module packages, module reloading, and the __name__ variable.Part VI, Classes and OOP Here, we explore Python’s object-oriented programming tool, the class—an op- tional but powerful way to structure code for customization and reuse. As you’ll see, classes mostly reuse ideas we will have covered by this point in the book, and OOP in Python is mostly about looking up names in linked objects. As you’ll also see, OOP is optional in Python, but it can shave development time substantially, especially for long-term strategic project development.Part VII, Exceptions and Tools We conclude the language fundamentals coverage in this text with a look at Py- thon’s exception handling model and statements, plus a brief overview of devel- opment tools that will become more useful when you start writing larger programs (debugging and testing tools, for instance). Although exceptions are a fairly light- weight tool, this part appears after the discussion of classes because exceptions should now all be classes.Part VIII, Advanced Topics (new in the fourth edition) In the final part, we explore some advanced topics. Here, we study Unicode and byte strings, managed attribute tools like properties and descriptors, function and class decorators, and metaclasses. These chapters are all optional reading, because not all programmers need to understand the subjects they address. On the other hand, readers who must process internationalized text or binary data, or are re- sponsible for developing APIs for other programmers to use, should find something of interest in this part.Part IX, Appendixes The book wraps up with a pair of appendixes that give platform-specific tips for using Python on various computers (Appendix A) and provide solutions to the end- of-part exercises (Appendix B). Solutions to end-of-chapter quizzes appear in the chapters themselves.xliv | Preface

www.it-ebooks.infoNote that the index and table of contents can be used to hunt for details, but there areno reference appendixes in this book (this book is a tutorial, not a reference). As men-tioned earlier, you can consult Python Pocket Reference, as well as other books, and thefree Python reference manuals maintained at http://www.python.org for syntax andbuilt-in tool details.Book UpdatesImprovements happen (and so do mis^H^H^H typos). Updates, supplements, and cor-rections for this book will be maintained (or referenced) on the Web at one of thefollowing sites: http://www.oreilly.com/catalog/9780596158064 (O’Reilly’s web page for the book) http://www.rmi.net/~lutz (the author’s site) http://www.rmi.net/~lutz/about-lp.html (the author’s web page for the book)The last of these three URLs points to a web page for this book where I will post updates,but be sure to search the Web if this link becomes invalid. If I could become moreclairvoyant, I would, but the Web changes faster than printed books.About the Programs in This BookThis fourth edition of this book, and all the program examples in it, is based on Pythonversion 3.0. In addition, most of its examples run under Python 2.6, as described in thetext, and notes for Python 2.6 readers are mixed in along the way.Because this text focuses on the core language, however, you can be fairly sure thatmost of what it has to say won’t change very much in future releases of Python. Mostof this book applies to earlier Python versions, too, except when it does not; naturally,if you try using extensions added after the release you’ve got, all bets are off.As a rule of thumb, the latest Python is the best Python. Because this book focuses onthe core language, most of it also applies to Jython, the Java-based Python languageimplementation, as well as other Python implementations described in Chapter 2.Source code for the book’s examples, as well as exercise solutions, can be fetched fromthe book’s website at http://www.oreilly.com/catalog/9780596158064/. So, how do yourun the examples? We’ll study startup details in Chapter 3, so please stay tuned forinformation on this front.Using Code ExamplesThis book is here to help you get your job done. In general, you may use the code inthis book in your programs and documentation. You do not need to contact us forpermission unless you’re reproducing a significant portion of the code. For example, Preface | xlv

www.it-ebooks.infowriting a program that uses several chunks of code from this book does not requirepermission. Selling or distributing a CD-ROM of examples from O’Reilly books doesrequire permission. Answering a question by citing this book and quoting examplecode does not require permission. Incorporating a significant amount of example codefrom this book into your product’s documentation does require permission.We appreciate, but do not require, attribution. An attribution usually includes the title,author, publisher, and ISBN. For example: “Learning Python, Fourth Edition, by MarkLutz. Copyright 2009 Mark Lutz, 978-0-596-15806-4.”If you feel your use of code examples falls outside fair use or the permission given above,feel free to contact us at [email protected] ConventionsThis book uses the following typographical conventions:Italic Used for email addresses, URLs, filenames, pathnames, and emphasizing new terms when they are first introducedConstant width Used for the contents of files and the output from commands, and to designate modules, methods, statements, and commandsConstant width bold Used in code sections to show commands or text that would be typed by the user, and, occasionally, to highlight portions of codeConstant width italic Used for replaceables and some comments in code sections<Constant width> Indicates a syntactic unit that should be replaced with real code Indicates a tip, suggestion, or general note relating to the nearby text. Indicates a warning or caution relating to the nearby text.xlvi | Preface

www.it-ebooks.info Notes specific to this book: In this book’s examples, the % character at the start of a system command line stands for the system’s prompt, whatever that may be on your machine (e.g., C:\Python30> in a DOS window). Don’t type the % character (or the system prompt it sometimes stands for) yourself. Similarly, in interpreter interaction listings, do not type the >>> and ... characters shown at the start of lines—these are prompts that Python displays. Type just the text after these prompts. To help you remember this, user inputs are shown in bold font in this book. Also, you normally don’t need to type text that starts with a # in listings; as you’ll learn, these are comments, not executable code.Safari® Books Online Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly.With a subscription, you can read any page and watch any video from our library online.Read books on your cell phone and mobile devices. Access new titles before they areavailable for print, and get exclusive access to manuscripts in development and postfeedback for the authors. Copy and paste code samples, organize your favorites, down-load chapters, bookmark key sections, create notes, print out pages, and benefit fromtons of other time-saving features.O’Reilly Media has uploaded this book to the Safari Books Online service. To have fulldigital access to this book and others on similar topics from O’Reilly and other pub-lishers, sign up for free at http://my.safaribooksonline.com.How to Contact UsPlease address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax)We will also maintain a web page for this book, where we list errata, examples, andany additional information. You can access this page at: http://www.oreilly.com/catalog/9780596158064/ Preface | xlvii

www.it-ebooks.infoTo comment or ask technical questions about this book, send email to: [email protected] more information about our books, conferences, Resource Centers, and theO’Reilly Network, see our website at: http://www.oreilly.comFor book updates, be sure to also see the other links mentioned earlier in this Preface.AcknowledgmentsAs I write this fourth edition of this book in 2009, I can’t help but be in a sort of “missionaccomplished” state of mind. I have now been using and promoting Python for 17 years,and have been teaching it for 12 years. Despite the passage of time and events, I am stillconstantly amazed at how successful Python has been over the years. It has grown inways that most of us could not possibly have imagined in 1992. So, at the risk ofsounding like a hopelessly self-absorbed author, you’ll have to pardon a few words ofreminiscing, congratulations, and thanks here.It’s been the proverbial long and winding road. Looking back today, when I first dis-covered Python in 1992, I had no idea what an impact it would have on the next 17years of my life. Two years after writing the first edition of Programming Python in1995, I began traveling around the country and the world teaching Python to beginnersand experts. Since finishing the first edition of Learning Python in 1999, I’ve been anindependent Python trainer and writer, thanks largely to Python’s exponential growthin popularity.As I write these words in mid-2009, I have written 12 Python books (4 editions of 3).I have also been teaching Python for more than a decade; have taught some 225 Pythontraining sessions in the U.S., Europe, Canada, and Mexico; and have met over 3,000students along the way. Besides racking up frequent flyer miles, these classes helpedme refine this text as well as my other Python books. Over the years, teaching honedthe books, and vice versa. In fact, the book you’re reading is derived almost entirelyfrom my classes.Because of this, I’d like to thank all the students who have participated in my coursesduring the last 12 years. Along with changes in Python itself, your feedback played ahuge role in shaping this text. (There’s nothing quite as instructive as watching 3,000students repeat the same beginner’s mistakes!) This edition owes its changes primarilyto classes held after 2003, though every class held since 1997 has in some way helpedrefine this book. I’d especially like to single out clients who hosted classes in Dublin,Mexico City, Barcelona, London, Edmonton, and Puerto Rico; better perks would behard to imagine.I’d also like to express my gratitude to everyone who played a part in producing thisbook. To the editors who worked on this project: Julie Steele on this edition, Tatianaxlviii | Preface

www.it-ebooks.infoApandi on the prior edition, and many others on earlier editions. To Doug Hellmannand Jesse Noller for taking part in the technical review of this book. And to O’Reillyfor giving me a chance to work on those 12 book projects—it’s been net fun (and onlyfeels a little like the movie Groundhog Day).I want to thank my original coauthor David Ascher as well for his work on the first twoeditions of this book. David contributed the “Outer Layers” part in prior editions,which we unfortunately had to trim to make room for new core language materials inthe third edition. To compensate, I added a handful of more advanced programs as aself-study final exercise in the third edition, and added both new advanced examplesand a new complete part for advanced topics in the fourth edition. Also see the priornotes in this Preface about follow-up application-level texts you may want to consultonce you’ve learned the fundamentals here.For creating such an enjoyable and useful language, I owe additional thanks to Guidovan Rossum and the rest of the Python community. Like most open source systems,Python is the product of many heroic efforts. After 17 years of programming Python, Istill find it to be seriously fun. It’s been my privilege to watch Python grow from a newkid on the scripting languages block to a widely used tool, deployed in some fashionby almost every organization writing software. That has been an exciting endeavor tobe a part of, and I’d like to thank and congratulate the entire Python community for ajob well done.I also want to thank my original editor at O’Reilly, the late Frank Willison. This bookwas largely Frank’s idea, and it reflects the contagious vision he had. In looking back,Frank had a profound impact on both my own career and that of Python itself. It is notan exaggeration to say that Frank was responsible for much of the fun and success ofPython when it was new. We still miss him.Finally, a few personal notes of thanks. To OQO for the best toys so far (while theylasted). To the late Carl Sagan for inspiring an 18-year-old kid from Wisconsin. To myMom, for courage. And to all the large corporations I’ve come across over the years,for reminding me how lucky I have been to be self-employed for the last decade!To my children, Mike, Sammy, and Roxy, for whatever futures you will choose to make.You were children when I began with Python, and you seem to have somehow grownup along the way; I’m proud of you. Life may compel us down paths all our own, butthere will always be a path home.And most of all, to Vera, my best friend, my girlfriend, and my wife. The best day ofmy life was the day I finally found you. I don’t know what the next 50 years hold, butI do know that I want to spend all of them holding you. —Mark Lutz Sarasota, Florida July 2009 Preface | xlix