Python Language Part 2

Python Language #python

exec statement is a function in Python 3 404 hasattr function bug in Python 2 405 Renamed modules 405 Compatibility 406 Octal Constants 406 All classes are \"new-style classes\" in Python 3. 406 Removed operators <> and ``, synonymous with != and repr() 407 encode/decode to hex no longer available 408 cmp function removed in Python 3 409 Leaked variables in list comprehension 409 map() 410 filter(), map() and zip() return iterators instead of sequences 411 Absolute/Relative Imports 412 More on Relative Imports 413 File I/O 414 The round() function tie-breaking and return type 414 round() tie breaking 414 round() return type 415 True, False and None 415 Return value when writing to a file object 416 long vs. int 416 Class Boolean Value 417 Chapter 80: Indentation 418 Examples 418 Indentation Errors 418 Simple example 418 Spaces or Tabs? 419 How Indentation is Parsed 419 Chapter 81: Indexing and Slicing 421 Syntax 421 Parameters 421 Remarks 421

Examples 421 Basic Slicing 421 Making a shallow copy of an array 422 Reversing an object 423 Indexing custom classes: __getitem__, __setitem__ and __delitem__ 423 Slice assignment 424 Slice objects 425 Basic Indexing 425 427 Chapter 82: Input, Subset and Output External Data Files using Pandas 427 Introduction 427 Examples 427 429 Basic Code to Import, Subset and Write External Data Files Using Pandas 429 429 Chapter 83: Introduction to RabbitMQ using AMQPStorm 429 Remarks 430 Examples 431 433 How to consume messages from RabbitMQ 433 How to publish messages to RabbitMQ 433 How to create a delayed queue in RabbitMQ 436 436 Chapter 84: IoT Programming with Python and Raspberry PI 436 Examples 437 437 Example - Temperature sensor 438 438 Chapter 85: Iterables and Iterators 438 Examples 439 439 Iterator vs Iterable vs Generator 439 What can be iterable Iterating over entire iterable Verify only one element in iterable Extract values one by one Iterator isn't reentrant! Chapter 86: Itertools Module Syntax Examples

Grouping items from an iterable object using a function 439 Take a slice of a generator 440 itertools.product 440 itertools.count 441 itertools.takewhile 442 itertools.dropwhile 443 Zipping two iterators until they are both exhausted 444 Combinations method in Itertools Module 444 Chaining multiple iterators together 445 itertools.repeat 445 Get an accumulated sum of numbers in an iterable 445 Cycle through elements in an iterator 446 itertools.permutations 446 Chapter 87: JSON Module 447 Remarks 447 447 Types 447 Defaults 447 De-serialisation types: 447 Serialisation types: 448 Custom (de-)serialisation 448 Serialisation: 448 De-serialisation: 449 Further custom (de-)serialisation: 449 Examples 449 Creating JSON from Python dict 449 Creating Python dict from JSON 450 Storing data in a file 450 Retrieving data from a file 450 `load` vs `loads`, `dump` vs `dumps` 451 Calling `json.tool` from the command line to pretty-print JSON output 452 Formatting JSON output 452 Setting indentation to get prettier output

Sorting keys alphabetically to get consistent output 452 Getting rid of whitespace to get compact output 453 453 JSON encoding custom objects 454 Chapter 88: kivy - Cross-platform Python Framework for NUI Development 454 454 Introduction Examples 454 First App 457 457 Chapter 89: Linked List Node Examples 457 Write a simple Linked List Node in python 458 458 Chapter 90: Linked lists 458 Introduction Examples 458 Single linked list example 462 462 Chapter 91: List 462 Introduction 462 Syntax 462 Remarks Examples 462 464 Accessing list values 469 List methods and supported operators 469 Length of a list 470 Iterating over a list 470 Checking whether an item is in a list 471 Reversing list elements 471 Checking if list is empty 472 Concatenate and Merge lists 473 Any and All 473 Remove duplicate values in list 475 Accessing values in nested list 475 Comparison of lists Initializing a List to a Fixed Number of Elements

Chapter 92: List comprehensions 476 Introduction 476 Syntax 476 Remarks 476 Examples 476 List Comprehensions 476 else 477 Double Iteration 478 In-place Mutation and Other Side Effects 478 Whitespace in list comprehensions 479 479 Dictionary Comprehensions 481 Generator Expressions 483 Use cases 483 Set Comprehensions 484 Avoid repetitive and expensive operations using conditional clause 486 Comprehensions involving tuples 486 Counting Occurrences Using Comprehension 487 Changing Types in a List 488 Chapter 93: List Comprehensions 488 Introduction 488 Syntax 488 Remarks 488 Examples 488 Conditional List Comprehensions 490 List Comprehensions with Nested Loops 491 Refactoring filter and map to list comprehensions 492 Refactoring - Quick Reference 492 Nested List Comprehensions 493 Iterate two or more list simultaneously within list comprehension 494 Chapter 94: List destructuring (aka packing and unpacking) 494 Examples

Destructuring assignment 494 Destructuring as values 494 Destructuring as a list 494 Ignoring values in destructuring assignments 495 Ignoring lists in destructuring assignments 495 495 Packing function arguments 496 Packing a list of arguments 496 Packing keyword arguments 498 Unpacking function arguments 499 499 Chapter 95: List slicing (selecting parts of lists) 499 Syntax 499 Remarks Examples 499 499 Using the third \"step\" argument 500 Selecting a sublist from a list 500 Reversing a list with slicing Shifting a list using slicing 502 502 Chapter 96: Logging Examples 502 503 Introduction to Python Logging Logging exceptions 506 506 Chapter 97: Loops 506 Introduction 506 Syntax 506 Parameters Examples 506 507 Iterating over lists For loops 508 508 Iterable objects and iterators 508 Break and Continue in Loops break statement

continue statement 509 Nested Loops 509 510 Use return from within a function as a break Loops with an \"else\" clause 510 512 Why would one use this strange construct? 513 Iterating over dictionaries 514 While Loop 515 The Pass Statement 515 Iterating different portion of a list with different step size 515 Iteration over the whole list 516 Iterate over sub-list 517 The \"half loop\" do-while 517 Looping and Unpacking 518 Chapter 98: Manipulating XML 518 Remarks 518 Examples 518 Opening and reading using an ElementTree 518 Modifying an XML File 519 Create and Build XML Documents 519 Opening and reading large XML files using iterparse (incremental parsing) 520 Searching the XML with XPath 522 Chapter 99: Map Function 522 Syntax 522 Parameters 522 Remarks 522 Examples 522 Basic use of map, itertools.imap and future_builtins.map 523 Mapping each value in an iterable 524 Mapping values of different iterables 525 Transposing with Map: Using \"None\" as function argument (python 2.x only) 526 Series and Parallel Mapping 529 Chapter 100: Math Module

Examples 529 Rounding: round, floor, ceil, trunc 529 Warning! 530 Warning about the floor, trunc, and integer division of negative numbers 530 Logarithms 530 Copying signs 531 Trigonometry 531 Calculating the length of the hypotenuse 531 Converting degrees to/from radians 531 Sine, cosine, tangent and inverse functions 531 Hyperbolic sine, cosine and tangent 532 Constants 532 Imaginary Numbers 533 Infinity and NaN (\"not a number\") 533 Pow for faster exponentiation 536 Complex numbers and the cmath module 536 540 Chapter 101: Metaclasses 540 Introduction 540 Remarks 540 Examples 540 541 Basic Metaclasses 542 Singletons using metaclasses 542 Using a metaclass 542 542 Metaclass syntax 543 Python 2 and 3 compatibility with six 543 543 Custom functionality with metaclasses 543 Introduction to Metaclasses 544 What is a metaclass? 546 The Simplest Metaclass A Metaclass which does Something The default metaclass Chapter 102: Method Overriding

Examples 546 Basic method overriding 546 547 Chapter 103: Mixins 547 Syntax 547 Remarks 547 Examples 547 548 Mixin 550 Overriding Methods in Mixins 550 550 Chapter 104: Multidimensional arrays 551 Examples 552 552 Lists in lists 552 Lists in lists in lists in... 553 554 Chapter 105: Multiprocessing 554 Examples 554 554 Running Two Simple Processes 555 Using Pool and Map 556 557 Chapter 106: Multithreading 557 Introduction 558 Examples 560 560 Basics of multithreading 560 Communicating between threads 560 Creating a worker pool 561 Advanced use of multithreads 561 Advanced printer (logger) Stoppable Thread with a while Loop Chapter 107: Mutable vs Immutable (and Hashable) in Python Examples Mutable vs Immutable Immutables Exercise Mutables

Exercise 562 Mutable and Immutable as Arguments 562 563 Exercise 564 Chapter 108: Neo4j and Cypher using Py2Neo 564 564 Examples 564 Importing and Authenticating 564 Adding Nodes to Neo4j Graph 565 Adding Relationships to Neo4j Graph 565 Query 1 : Autocomplete on News Titles 565 Query 2 : Get News Articles by Location on a particular date 567 Cypher Query Samples 567 567 Chapter 109: Non-official Python implementations 567 Examples 567 567 IronPython 568 568 Hello World 568 External links 568 568 Jython 569 Hello World 569 External links 570 571 Transcrypt 572 Code size and speed 572 Integration with HTML 572 Integration with JavaScript and DOM 572 Integration with other JavaScript libraries 572 Relation between Python and JavaScript code External links Chapter 110: Operator module Examples Operators as alternative to an infix operator Methodcaller Itemgetter

Chapter 111: Operator Precedence 574 Introduction 574 Remarks 574 Examples 575 Simple Operator Precedence Examples in python. 575 Chapter 112: Optical Character Recognition 576 Introduction 576 Examples 576 PyTesseract 576 PyOCR 576 Chapter 113: os.path 578 Introduction 578 Syntax 578 Examples 578 Join Paths 578 Absolute Path from Relative Path 578 Path Component Manipulation 579 Get the parent directory 579 If the given path exists. 579 check if the given path is a directory, file, symbolic link, mount point etc. 579 Chapter 114: Overloading 581 Examples 581 Magic/Dunder Methods 581 Container and sequence types 582 Callable types 583 Handling unimplemented behaviour 583 Operator overloading 584 Chapter 115: Pandas Transform: Preform operations on groups and concatenate the results 587 Examples 587 Simple transform 587 First, Lets create a dummy dataframe 587 Now, we will use pandas transform function to count the number of orders per customer 587

Multiple results per group 588 Using transform functions that return sub-calculations per group 588 Chapter 116: Parallel computation 590 590 Remarks 590 Examples 590 Using the multiprocessing module to parallelise tasks 590 Using Parent and Children scripts to execute code in parallel 591 Using a C-extension to parallelize tasks 591 Using PyPar module to parallelize 593 Chapter 117: Parsing Command Line arguments 593 Introduction 593 Examples 593 Hello world in argparse 594 Basic example with docopt 594 Setting mutually exclusive arguments with argparse 595 Using command line arguments with argv 596 Custom parser error message with argparse 596 Conceptual grouping of arguments with argparse.add_argument_group() 598 Advanced example with docopt and docopt_dispatch 599 Chapter 118: Partial functions 599 Introduction 599 Syntax 599 Parameters 599 Remarks 599 Examples 599 Raise the power 601 Chapter 119: Performance optimization 601 Remarks 601 Examples 601 Code profiling 604 Chapter 120: Pickle data serialisation 604 Syntax

Parameters 604 Remarks 604 Pickleable types 604 pickle and security 604 Examples 605 605 Using Pickle to serialize and deserialize an object 605 605 To serialize the object 605 To deserialize the object 606 Using pickle and byte objects 608 608 Customize Pickled Data 608 Chapter 121: Pillow 608 609 Examples 609 Read Image File 609 Convert files to JPEG 609 610 Chapter 122: pip: PyPI Package Manager 610 Introduction 610 Syntax 610 Remarks 610 Examples 611 611 Install Packages 611 612 Install from requirements files 612 Uninstall Packages 612 To list all packages installed using `pip` 613 Upgrade Packages 614 Updating all outdated packages on Linux Updating all outdated packages on Windows Create a requirements.txt file of all packages on the system Create a requirements.txt file of packages only in the current virtualenv Using a certain Python version with pip Installing packages not yet on pip as wheels Note on Installing Pre-Releases

Note on Installing Development Versions 614 Chapter 123: Plotting with Matplotlib 617 617 Introduction 617 Examples 617 618 A Simple Plot in Matplotlib 619 Adding more features to a simple plot : axis labels, title, axis ticks, grid, and legend 620 Making multiple plots in the same figure by superimposition similar to MATLAB 621 Making multiple Plots in the same figure using plot superimposition with separate plot com 623 Plots with Common X-axis but different Y-axis : Using twinx() 626 Plots with common Y-axis and different X-axis using twiny() 626 626 Chapter 124: Plugin and Extension Classes 627 Examples 629 629 Mixins 629 Plugins with Customized Classes 631 633 Chapter 125: Polymorphism 633 Examples 633 633 Basic Polymorphism 633 Duck Typing 635 635 Chapter 126: PostgreSQL 635 Examples 635 637 Getting Started 637 Installation using pip 637 Basic usage 638 640 Chapter 127: Processes and Threads Introduction Examples Global Interpreter Lock Running in Multiple Threads Running in Multiple Processes Sharing State Between Threads Sharing State Between Processes Chapter 128: Profiling

Examples 640 %%timeit and %timeit in IPython 640 timeit() function 640 timeit command line 640 line_profiler in command line 641 Using cProfile (Preferred Profiler) 641 643 Chapter 129: Property Objects 643 Remarks 643 Examples 643 643 Using the @property decorator 644 Using the @property decorator for read-write properties 644 Overriding just a getter, setter or a deleter of a property object 647 Using properties without decorators 647 647 Chapter 130: py.test 647 Examples 647 647 Setting up py.test 648 The Code to Test 648 The Testing Code 649 Running The Test 651 653 Failing Tests 653 Intro to Test Fixtures 653 py.test fixtures to the rescue! 653 Cleaning up after the tests are done. 653 654 Chapter 131: pyaudio 656 Introduction 656 Remarks 656 Examples Callback Mode Audio I/O Blocking Mode Audio I/O Chapter 132: pyautogui module Introduction Examples

Mouse Functions 656 Keyboard Functions 656 ScreenShot And Image Recognition 656 Chapter 133: pygame 657 Introduction 657 Syntax 657 Parameters 657 Examples 657 Installing pygame 657 Pygame's mixer module 658 Initializing 658 Possible Actions 658 Channels 658 Chapter 134: Pyglet 660 660 Introduction 660 Examples 660 Hello World in Pyglet 660 Installation of Pyglet 660 Playing Sound in Pyglet 660 Using Pyglet for OpenGL 661 Drawing Points Using Pyglet and OpenGL 662 Chapter 135: PyInstaller - Distributing Python Code 662 Syntax 662 Remarks 662 Examples 662 Installation and Setup 663 Using Pyinstaller 663 Bundling to One Folder 663 Advantages: 663 Disadvantages 664 Bundling to a Single File

Chapter 136: Python and Excel 665 Examples 665 Put list data into a Excel's file. 665 OpenPyXL 665 Create excel charts with xlsxwriter 666 Read the excel data using xlrd module 668 Format Excel files with xlsxwriter 669 Chapter 137: Python Anti-Patterns 671 Examples 671 Overzealous except clause 671 Looking before you leap with processor-intensive function 672 Dictionary keys 672 Chapter 138: Python concurrency 674 Remarks 674 Examples 674 The threading module 674 The multiprocessing module 674 Passing data between multiprocessing processes 675 Chapter 139: Python Data Types 677 Introduction 677 Examples 677 Numbers data type 677 String Data Type 677 List Data Type 677 Tuple Data Type 677 Dictionary Data Type 678 Set Data Types 678 Chapter 140: Python HTTP Server 679 Examples 679 Running a simple HTTP server 679 Serving files 679 Programmatic API of SimpleHTTPServer 681

Basic handling of GET, POST, PUT using BaseHTTPRequestHandler 682 Chapter 141: Python Lex-Yacc 684 Introduction 684 Remarks 684 Examples 684 Getting Started with PLY 684 The \"Hello, World!\" of PLY - A Simple Calculator 684 Part 1: Tokenizing Input with Lex 686 Breakdown 687 h22 688 h23 688 h24 688 h25 689 h26 689 h27 689 h28 689 h29 689 h210 689 h211 690 Part 2: Parsing Tokenized Input with Yacc 690 Breakdown 691 h212 692 694 Chapter 142: Python Networking 694 Remarks 694 Examples 694 The simplest Python socket client-server example 694 Creating a Simple Http Server 695 Creating a TCP server 696 Creating a UDP Server 696 Start Simple HttpServer in a thread and open the browser 698 Chapter 143: Python Persistence

Syntax 698 Parameters 698 Examples 698 698 Python Persistence 699 Function utility for save and load 700 700 Chapter 144: Python Requests Post 700 Introduction 700 Examples 701 702 Simple Post 702 Form Encoded Data 703 File Upload 704 Responses 705 Authentication 705 Proxies 705 705 Chapter 145: Python Serial Communication (pyserial) 705 Syntax 705 Parameters 705 Remarks 706 Examples 707 707 Initialize serial device 707 Read from serial port 707 Check what serial ports are available on your machine 707 708 Chapter 146: Python Server Sent Events 708 Introduction 708 Examples 708 709 Flask SSE Asyncio SSE Chapter 147: Python speed of program Examples Notation List operations Deque operations

Set operations 710 Algorithmic Notations... 710 Chapter 148: Python Virtual Environment - virtualenv 712 Introduction 712 Examples 712 Installation 712 Usage 712 Install a package in your Virtualenv 713 Other useful virtualenv commands 713 Chapter 149: Queue Module 714 Introduction 714 Examples 714 Simple example 714 Chapter 150: Raise Custom Errors / Exceptions 715 Introduction 715 Examples 715 Custom Exception 715 Catch custom Exception 715 Chapter 151: Random module 717 Syntax 717 Examples 717 Random and sequences: shuffle, choice and sample 717 shuffle() 717 choice() 717 sample() 717 718 Creating random integers and floats: randint, randrange, random, and uniform 718 randint() 718 randrange() 719 random 719 uniform 719 Reproducible random numbers: Seed and State

Create cryptographically secure random numbers 720 Creating a random user password 721 Random Binary Decision 722 Chapter 152: Reading and Writing CSV 723 Examples 723 723 Writing a TSV file 723 723 Python 723 Output file 724 724 Using pandas 724 Chapter 153: Recursion 724 724 Remarks 728 Examples 729 729 Sum of numbers from 1 to n 730 The What, How, and When of Recursion 732 Tree exploration with recursion 732 Increasing the Maximum Recursion Depth 732 Tail Recursion - Bad Practice 732 Tail Recursion Optimization Through Stack Introspection 732 732 Chapter 154: Reduce 733 Syntax 734 Parameters 734 Remarks 734 Examples 735 735 Overview 735 Using reduce Cumulative product Non short-circuit variant of any/all First truthy/falsy element of a sequence (or last element if there is none) Chapter 155: Regular Expressions (Regex) Introduction Syntax

Examples 735 Matching the beginning of a string 735 Searching 736 Grouping 737 738 Named groups 738 Non-capturing groups 738 739 Escaping Special Characters 739 Replacing 739 Replacing strings 740 Using group references 740 Using a replacement function 740 Find All Non-Overlapping Matches 741 Precompiled patterns 741 Checking for allowed characters 741 Splitting a string using regular expressions 741 Flags 742 Flags keyword 742 Inline flags 743 745 Iterating over matches using `re.finditer` 745 Match an expression only in specific locations 745 745 Chapter 156: Searching 745 Remarks 745 Examples 746 746 Getting the index for strings: str.index(), str.rindex() and str.find(), str.rfind() 746 Searching for an element 746 List 746 Tuple 747 String Set Dict Getting the index list and tuples: list.index(), tuple.index() Searching key(s) for a value in dict

Getting the index for sorted sequences: bisect.bisect_left() 747 Searching nested sequences 748 Searching in custom classes: __contains__ and __iter__ 749 Chapter 157: Secure Shell Connection in Python 750 Parameters 750 Examples 750 ssh connection 750 Chapter 158: Security and Cryptography 751 Introduction 751 Syntax 751 Remarks 751 Examples 751 Calculating a Message Digest 751 Available Hashing Algorithms 752 Secure Password Hashing 752 File Hashing 752 Symmetric encryption using pycrypto 753 Generating RSA signatures using pycrypto 754 Asymmetric RSA encryption using pycrypto 755 Chapter 159: Set 756 Syntax 756 Remarks 756 Examples 756 Get the unique elements of a list 756 Operations on sets 757 Sets versus multisets 758 Set Operations using Methods and Builtins 759 Intersection 759 Union 759 Difference 759 Symmetric Difference 759

Subset and superset 760 Disjoint sets 760 Testing membership 760 Length 761 761 Set of Sets 762 Chapter 160: setup.py 762 762 Parameters 762 Remarks Examples 762 763 Purpose of setup.py 763 Adding command line scripts to your python package 764 Using source control metadata in setup.py Adding installation options 766 766 Chapter 161: shelve 766 Introduction 766 Remarks 766 Warning: 766 Restrictions 766 Examples 767 767 Sample code for shelve 768 To summarize the interface (key is a string, data is an arbitrary object): Creating a new Shelf 770 Write-back 770 770 Chapter 162: Similarities in syntax, Differences in meaning: Python vs. JavaScript Introduction 770 Examples 771 `in` with lists 771 771 Chapter 163: Simple Mathematical Operators 771 Introduction Remarks Numerical types and their metaclasses

Examples 771 Addition 771 Subtraction 772 Multiplication 772 Division 773 Exponentation 775 775 Special functions 776 Logarithms 776 Inplace Operations 777 Trigonometric Functions 777 Modulus 779 779 Chapter 164: Sockets 779 Introduction 779 Parameters 779 Examples 779 780 Sending data via UDP 780 Receiving data via UDP 782 Sending data via TCP 783 Multi-threaded TCP Socket Server 783 Raw Sockets on Linux 783 786 Chapter 165: Sockets And Message Encryption/Decryption Between Client and Server 786 Introduction 788 Remarks 791 Examples 791 791 Server side Implementation 791 Client side Implementation 791 792 Chapter 166: Sorting, Minimum and Maximum Examples Getting the minimum or maximum of several values Using the key argument Default Argument to max, min Special case: dictionaries

By value 792 Getting a sorted sequence 793 Minimum and Maximum of a sequence 793 Make custom classes orderable 794 Extracting N largest or N smallest items from an iterable 796 798 Chapter 167: Sqlite3 Module 798 Examples 798 Sqlite3 - Not require separate server process. 798 Getting the values from the database and Error handling 800 Chapter 168: Stack 800 Introduction 800 Syntax 800 Remarks 800 Examples 800 Creating a Stack class with a List Object 801 Parsing Parentheses 803 Chapter 169: String Formatting 803 Introduction 803 Syntax 803 Remarks 803 Examples 803 Basics of String Formatting 805 Alignment and padding 805 Format literals (f-string) 806 String formatting with datetime 806 Format using Getitem and Getattr 807 Float formatting 808 Formatting Numerical Values 808 Custom formatting for a class 809 Nested formatting 810 Padding and truncating strings, combined 811 Named placeholders

Using a dictionary (Python 2.x) 811 Using a dictionary (Python 3.2+) 811 Without a dictionary: 811 Chapter 170: String Methods 812 Syntax 812 Remarks 813 Examples 813 Changing the capitalization of a string 813 str.casefold() 813 str.upper() 813 str.lower() 814 str.capitalize() 814 str.title() 814 str.swapcase() 814 Usage as str class methods 814 Split a string based on a delimiter into a list of strings 815 str.split(sep=None, maxsplit=-1) 815 str.rsplit(sep=None, maxsplit=-1) 816 Replace all occurrences of one substring with another substring 816 str.replace(old, new[, count]): 816 str.format and f-strings: Format values into a string 817 Counting number of times a substring appears in a string 818 str.count(sub[, start[, end]]) 818 Test the starting and ending characters of a string 819 str.startswith(prefix[, start[, end]]) 819 str.endswith(prefix[, start[, end]]) 819 Testing what a string is composed of 820 str.isalpha 820 str.isupper, str.islower, str.istitle 820 str.isdecimal, str.isdigit, str.isnumeric 821 str.isalnum 821 str.isspace 822 str.translate: Translating characters in a string 822

Stripping unwanted leading/trailing characters from a string 823 str.strip([chars]) 823 str.rstrip([chars]) and str.lstrip([chars]) 823 Case insensitive string comparisons 824 Join a list of strings into one string 825 String module's useful constants 825 string.ascii_letters: 825 string.ascii_lowercase: 826 string.ascii_uppercase: 826 string.digits: 826 string.hexdigits: 826 string.octaldigits: 826 string.punctuation: 826 string.whitespace: 826 string.printable: 827 Reversing a string 827 Justify strings 827 Conversion between str or bytes data and unicode characters 828 String Contains 829 Chapter 171: String representations of class instances: __str__ and __repr__ methods 830 Remarks 830 830 A note about implemeting both methods 830 Notes 830 Examples 831 Motivation 832 The Problem 832 The Solution (Part 1) 833 The Solution (Part 2) 834 About those duplicated functions... 835 Summary 835 837 Both methods implemented, eval-round-trip style __repr__() Chapter 172: Subprocess Library

Syntax 837 Parameters 837 Examples 837 837 Calling External Commands 837 More flexibility with Popen 838 838 Launching a subprocess 838 Waiting on a subprocess to complete 838 Reading output from a subprocess 838 Interactive access to running subprocesses 839 839 Writing to a subprocess 840 Reading a stream from a subprocess 840 840 How to create the command list argument 840 840 Chapter 173: sys 840 Introduction 840 Syntax 841 Remarks 841 Examples 842 842 Command line arguments 842 Script name 842 Standard error stream 844 Ending the process prematurely and returning an exit code 844 844 Chapter 174: tempfile NamedTemporaryFile 844 Parameters 845 Examples 845 Create (and write to a) known, persistant temporary file Chapter 175: Templates in python Examples Simple data output program using template Changing delimiter Chapter 176: The __name__ special variable Introduction

Remarks 845 Examples 845 845 __name__ == '__main__' 845 Situation 1 845 Situation 2 846 function_class_or_module.__name__ 847 Use in logging 848 848 Chapter 177: The base64 Module 848 Introduction 848 Syntax 850 Parameters 850 Remarks 850 Examples 852 852 Encoding and Decoding Base64 853 Encoding and Decoding Base32 853 Encoding and Decoding Base16 855 Encoding and Decoding ASCII85 855 Encoding and Decoding Base85 855 855 Chapter 178: The dis module 855 Examples 857 857 Constants in the dis module 857 What is Python bytecode? 857 Disassembling modules 858 858 Chapter 179: The Interpreter (Command Line Console) 858 Examples 859 861 Getting general help Referring to the last expression Opening the Python console The PYTHONSTARTUP variable Command line arguments Getting help about an object Chapter 180: The locale Module

Remarks 861 Examples 861 861 Currency Formatting US Dollars Using the locale Module 862 862 Chapter 181: The os Module 862 Introduction 862 Syntax 862 Parameters 862 Examples 862 862 Create a directory 863 Get current directory 863 Determine the name of the operating system 863 Remove a directory 863 Follow a symlink (POSIX) 865 Change permissions on a file 865 makedirs - recursive directory creation 865 867 Chapter 182: The pass statement 867 Syntax 867 Remarks 868 Examples 868 868 Ignore an exception 869 Create a new Exception that can be caught 871 871 Chapter 183: The Print Function 871 Examples 871 871 Print basics 872 Print parameters 872 Chapter 184: tkinter Introduction Remarks Examples A minimal tkinter Application Geometry Managers Place

Pack 873 Grid 873 Chapter 185: Tuple 875 875 Introduction 875 Syntax 875 Remarks 875 Examples 875 Indexing Tuples 876 Tuples are immutable 876 Tuple Are Element-wise Hashable and Equatable 877 Tuple 878 Packing and Unpacking Tuples 879 Reversing Elements 879 Built-in Tuple Functions 879 Comparison 879 Tuple Length 880 Max of a tuple 880 Min of a tuple 880 Convert a list into tuple 880 Tuple concatenation 881 Chapter 186: Turtle Graphics 881 Examples 881 Ninja Twist (Turtle Graphics) 882 Chapter 187: Type Hints 882 Syntax 882 Remarks 882 Examples 882 Generic Types 882 Adding types to a function 883 Class Members and Methods 884 Variables and Attributes

NamedTuple 885 Type hints for keyword arguments 885 Chapter 188: Unicode 886 Examples 886 Encoding and decoding 886 Chapter 189: Unicode and bytes 887 Syntax 887 Parameters 887 Examples 887 Basics 887 Unicode to bytes 887 Bytes to unicode 888 888 Encoding/decoding error handling 889 Encoding 889 Decoding 889 Morale 889 891 File I/O 891 Chapter 190: Unit Testing 891 Remarks 891 Examples 892 893 Testing Exceptions 894 Mocking functions with unittest.mock.create_autospec 895 Test Setup and Teardown within a unittest.TestCase 896 Asserting on Exceptions Choosing Assertions Within Unittests 900 Unit tests with pytest 900 900 Chapter 191: Unzipping Files Introduction 900 Examples 900 Using Python ZipFile.extractall() to decompress a ZIP file Using Python TarFile.extractall() to decompress a tarball

Chapter 192: urllib 901 Examples 901 HTTP GET 901 Python 2 901 Python 3 901 901 HTTP POST 902 Python 2 902 Python 3 902 Decode received bytes according to content type encoding 904 904 Chapter 193: Usage of \"pip\" module: PyPI Package Manager 904 Introduction 905 Syntax Examples 905 905 Example use of commands 906 Handling ImportError Exception Force install 907 907 Chapter 194: User-Defined Methods Examples 907 908 Creating user-defined method objects Turtle example 909 909 Chapter 195: Using loops within functions 909 Introduction Examples 909 Return statement inside loop in a function 910 910 Chapter 196: Variable Scope and Binding 910 Syntax Examples 910 911 Global Variables 912 Local Variables 912 Nonlocal Variables 913 Binding Occurrence Functions skip class scope when looking up names

The del command 914 del v 914 del v.name 914 del v[item] 914 del v[a:b] 914 915 Local vs Global Scope 915 What are local and global scope? 915 916 What happens with name clashes? 917 Functions within functions 919 global vs nonlocal (Python 3 only) 919 Chapter 197: virtual environment with virtualenvwrapper 919 Introduction 919 Examples 921 Create virtual environment with virtualenvwrapper 921 921 Chapter 198: Virtual environments 921 Introduction Remarks 921 Examples 921 Creating and using a virtual environment 921 922 Installing the virtualenv tool 922 Creating a new virtual environment 922 Activating an existing virtual environment 923 923 Saving and restoring dependencies 923 Exiting a virtual environment 925 Using a virtual environment in a shared host 925 Built-in virtual environments 925 Installing packages in a virtual environment Creating a virtual environment for a different version of python Managing multiple virtual enviroments with virtualenvwrapper Installation

Usage 926 Project Directories 926 927 Discovering which virtual environment you are using 927 Specifying specific python version to use in script on Unix/Linux 927 Using virtualenv with fish shell 928 Making virtual environments using Anaconda 928 Create an environment 929 Activate and deactivate your environment 929 View a list of created environments 929 Remove an environment 929 Checking if running inside a virtual environment 930 930 Chapter 199: Web scraping with Python 930 Introduction 930 Remarks 930 930 Useful Python packages for web scraping (alphabetical order) 930 Making requests and collecting data 930 930 requests 930 requests-cache 930 scrapy 931 selenium 931 HTML parsing 931 BeautifulSoup 931 lxml 931 Examples 932 Basic example of using requests and lxml to scrape some data 933 Maintaining web-scraping session with requests 933 Scraping using the Scrapy framework 933 Modify Scrapy user agent 934 Scraping using BeautifulSoup4 Scraping using Selenium WebDriver Simple web content download with urllib.request Scraping with curl

Chapter 200: Web Server Gateway Interface (WSGI) 935 Parameters 935 Examples 935 Server Object (Method) 935 Chapter 201: Webbrowser Module 937 Introduction 937 Syntax 937 Parameters 937 Remarks 938 Examples 939 Opening a URL with Default Browser 939 Opening a URL with Different Browsers 939 Chapter 202: Websockets 941 Examples 941 Simple Echo with aiohttp 941 Wrapper Class with aiohttp 941 Using Autobahn as a Websocket Factory 942 Chapter 203: Working around the Global Interpreter Lock (GIL) 944 Remarks 944 944 Why is there a GIL? 944 Details on how the GIL operates: 944 Benefits of the GIL 945 Consequences of the GIL 945 References: 945 Examples 945 Multiprocessing.Pool 946 David Beazley's code that showed GIL threading problems 947 Cython nogil: 947 947 David Beazley's code that showed GIL threading problems 949 Re-written using nogil (ONLY WORKS IN CYTHON): Chapter 204: Working with ZIP archives

Syntax 949 Remarks 949 Examples 949 949 Opening Zip Files 949 Examining Zipfile Contents 950 Extracting zip file contents to a directory 950 Creating new archives 952 952 Chapter 205: Writing extensions 952 Examples 953 953 Hello World with C Extension 953 Passing an open file to C Extensions 955 C Extension Using c++ and Boost 955 955 C++ Code 955 Chapter 206: Writing to CSV from String or List 955 955 Introduction 956 Parameters 957 Remarks Examples Basic Write Example Appending a String as a newline in a CSV file Credits

Chapter 80: Indentation Examples Indentation Errors The spacing should be even and uniform throughout. Improper indentation can cause an IndentationError or cause the program to do something unexpected. The following example raises an IndentationError: a=7 if a > 5: print \"foo\" else: print \"bar\" print \"done\" Or if the line following a colon is not indented, an IndentationError will also be raised: if True: print \"true\" If you add indentation where it doesn't belong, an IndentationError will be raised: if True: a=6 b=5 If you forget to un-indent functionality could be lost. In this example None is returned instead of the expected False: def isEven(a): if a%2 ==0: return True #this next line should be even with the if return False print isEven(7) Simple example For Python, Guido van Rossum based the grouping of statements on indentation. The reasons for this are explained in the first section of the \"Design and History Python FAQ\". Colons, :, are used to declare an indented code block, such as the following example: class ExampleClass: #Every function belonging to a class must be indented equally def __init__(self): name = \"example\" https://riptutorial.com/ 418

def someFunction(self, a): #Notice everything belonging to a function must be indented if a > 5: return True else: return False #If a function is not indented to the same level it will not be considers as part of the parent class def separateFunction(b): for i in b: #Loops are also indented and nested conditions start a new indentation if i == 1: return True return False separateFunction([2,3,5,6,1]) Spaces or Tabs? The recommended indentation is 4 spaces but tabs or spaces can be used so long as they are consistent. Do not mix tabs and spaces in Python as this will cause an error in Python 3 and can causes errors in Python 2. How Indentation is Parsed Whitespace is handled by the lexical analyzer before being parsed. The lexical analyzer uses a stack to store indentation levels. At the beginning, the stack contains just the value 0, which is the leftmost position. Whenever a nested block begins, the new indentation level is pushed on the stack, and an \"INDENT\" token is inserted into the token stream which is passed to the parser. There can never be more than one \"INDENT\" token in a row ( IndentationError). When a line is encountered with a smaller indentation level, values are popped from the stack until a value is on top which is equal to the new indentation level (if none is found, a syntax error occurs). For each value popped, a \"DEDENT\" token is generated. Obviously, there can be multiple \"DEDENT\" tokens in a row. The lexical analyzer skips empty lines (those containing only whitespace and possibly comments), and will never generate either \"INDENT\" or \"DEDENT\" tokens for them. At the end of the source code, \"DEDENT\" tokens are generated for each indentation level left on the stack, until just the 0 is left. For example: if foo: if bar: x = 42 else: https://riptutorial.com/ 419

print foo is analyzed as: <if> <foo> <:> [0] <INDENT> <if> <bar> <:> [0, 4] <INDENT> <x> <=> <42> [0, 4, 8] <DEDENT> <DEDENT> <else> <:> [0] <INDENT> <print> <foo> [0, 2] <DEDENT> The parser than handles the \"INDENT\" and \"DEDENT\" tokens as block delimiters. Read Indentation online: https://riptutorial.com/python/topic/2597/indentation https://riptutorial.com/ 420

Chapter 81: Indexing and Slicing Syntax • obj[start:stop:step] • slice(stop) • slice(start, stop[, step]) Parameters Paramer Description obj The object that you want to extract a \"sub-object\" from start The index of obj that you want the sub-object to start from (keep in mind that Python is zero-indexed, meaning that the first item of obj has an index of 0). If omitted, defaults to 0. The (non-inclusive) index of obj that you want the sub-object to end at. If omitted, stop defaults to len(obj). step Allows you to select only every step item. If omitted, defaults to 1. Remarks You can unify the concept of slicing strings with that of slicing other sequences by viewing strings as an immutable collection of characters, with the caveat that a unicode character is represented by a string of length 1. In mathematical notation you can consider slicing to use a half-open interval of [start, end), that is to say that the start is included but the end is not. The half-open nature of the interval has the advantage that len(x[:n]) = n where len(x) > =n, while the interval being closed at the start has the advantage that x[n:n+1] = [x[n]] where x is a list with len(x) >= n, thus keeping consistency between indexing and slicing notation. Examples Basic Slicing For any iterable (for eg. a string, list, etc), Python allows you to slice and return a substring or sublist of its data. Format for slicing: https://riptutorial.com/ 421

iterable_name[start:stop:step] where, • start is the first index of the slice. Defaults to 0 (the index of the first element) • stop one past the last index of the slice. Defaults to len(iterable) • step is the step size (better explained by the examples below) Examples: a = \"abcdef\" a # \"abcdef\" # Same as a[:] or a[::] since it uses the defaults for all three indices a[-1] # \"f\" a[:] # \"abcdef\" a[::] # \"abcdef\" a[3:] # \"def\" (from index 3, to end(defaults to size of iterable)) a[:4] # \"abcd\" (from beginning(default 0) to position 4 (excluded)) a[2:4] # \"cd\" (from position 2, to position 4 (excluded)) In addition, any of the above can be used with the step size defined: a[::2] # \"ace\" (every 2nd element) a[1:4:2] # \"bd\" (from index 1, to index 4 (excluded), every 2nd element) Indices can be negative, in which case they're computed from the end of the sequence a[:-1] # \"abcde\" (from index 0 (default), to the second last element (last element - 1)) a[:-2] # \"abcd\" (from index 0 (default), to the third last element (last element -2)) a[-1:] # \"f\" (from the last element to the end (default len()) Step sizes can also be negative, in which case slice will iterate through the list in reverse order: a[3:1:-1] # \"dc\" (from index 2 to None (default), in reverse order) This construct is useful for reversing an iterable a[::-1] # \"fedcba\" (from last element (default len()-1), to first, in reverse order(-1)) Notice that for negative steps the default end_index is None (see http://stackoverflow.com/a/12521981 ) a[5:None:-1] # \"fedcba\" (this is equivalent to a[::-1]) a[5:0:-1] # \"fedcb\" (from the last element (index 5) to second element (index 1) Making a shallow copy of an array A quick way to make a copy of an array (as opposed to assigning a variable with another reference to the original array) is: https://riptutorial.com/ 422

arr[:] Let's examine the syntax. [:] means that start, end, and slice are all omitted. They default to 0, len(arr), and 1, respectively, meaning that subarray that we are requesting will have all of the elements of arr from the beginning until the very end. In practice, this looks something like: arr = ['a', 'b', 'c'] copy = arr[:] arr.append('d') print(arr) # ['a', 'b', 'c', 'd'] print(copy) # ['a', 'b', 'c'] As you can see, arr.append('d') added d to arr, but copy remained unchanged! Note that this makes a shallow copy, and is identical to arr.copy(). Reversing an object You can use slices to very easily reverse a str, list, or tuple (or basically any collection object that implements slicing with the step parameter). Here is an example of reversing a string, although this applies equally to the other types listed above: s = 'reverse me!' s[::-1] # '!em esrever' Let's quickly look at the syntax. [::-1] means that the slice should be from the beginning until the end of the string (because start and end are omitted) and a step of -1 means that it should move through the string in reverse. Indexing custom classes: __getitem__, __setitem__ and __delitem__ class MultiIndexingList: def __init__(self, value): self.value = value def __repr__(self): return repr(self.value) def __getitem__(self, item): if isinstance(item, (int, slice)): return self.__class__(self.value[item]) return [self.value[i] for i in item] def __setitem__(self, item, value): if isinstance(item, int): self.value[item] = value elif isinstance(item, slice): raise ValueError('Cannot interpret slice with multiindexing') else: for i in item: if isinstance(i, slice): https://riptutorial.com/ 423

raise ValueError('Cannot interpret slice with multiindexing') self.value[i] = value def __delitem__(self, item): if isinstance(item, int): del self.value[item] elif isinstance(item, slice): del self.value[item] else: if any(isinstance(elem, slice) for elem in item): raise ValueError('Cannot interpret slice with multiindexing') item = sorted(item, reverse=True) for elem in item: del self.value[elem] This allows slicing and indexing for element access: a = MultiIndexingList([1,2,3,4,5,6,7,8]) <-- indicated which element came from which index a # Out: [1, 2, 3, 4, 5, 6, 7, 8] a[1,5,2,6,1] # Out: [2, 6, 3, 7, 2] a[4, 1, 5:, 2, ::2] # Out: [5, 2, [6, 7, 8], 3, [1, 3, 5, 7]] # 4|1-|----50:---|2-|-----::2----- While setting and deleting elements only allows for comma seperated integer indexing (no slicing): a[4] = 1000 a # Out: [1, 2, 3, 4, 1000, 6, 7, 8] a[2,6,1] = 100 a # Out: [1, 100, 100, 4, 1000, 6, 100, 8] del a[5] a # Out: [1, 100, 100, 4, 1000, 100, 8] del a[4,2,5] a # Out: [1, 100, 4, 8] Slice assignment Another neat feature using slices is slice assignment. Python allows you to assign new slices to replace old slices of a list in a single operation. This means that if you have a list, you can replace multiple members in a single assignment: lst = [1, 2, 3] lst[1:3] = [4, 5] print(lst) # Out: [1, 4, 5] The assignment shouldn't match in size as well, so if you wanted to replace an old slice with a new slice that is different in size, you could: https://riptutorial.com/ 424

lst = [1, 2, 3, 4, 5] lst[1:4] = [6] print(lst) # Out: [1, 6, 5] It's also possible to use the known slicing syntax to do things like replacing the entire list: lst = [1, 2, 3] lst[:] = [4, 5, 6] print(lst) # Out: [4, 5, 6] Or just the last two members: lst = [1, 2, 3] lst[-2:] = [4, 5, 6] print(lst) # Out: [1, 4, 5, 6] Slice objects Slices are objects in themselves and can be stored in variables with the built-in slice() function. Slice variables can be used to make your code more readable and to promote reuse. >>> programmer_1 = [ 1956, 'Guido', 'van Rossum', 'Python', 'Netherlands'] >>> programmer_2 = [ 1815, 'Ada', 'Lovelace', 'Analytical Engine', 'England'] >>> name_columns = slice(1, 3) >>> programmer_1[name_columns] ['Guido', 'van Rossum'] >>> programmer_2[name_columns] ['Ada', 'Lovelace'] Basic Indexing Python lists are 0-based i.e. the first element in the list can be accessed by the index 0 arr = ['a', 'b', 'c', 'd'] print(arr[0]) >> 'a' You can access the second element in the list by index 1, third element by index 2 and so on: print(arr[1]) >> 'b' print(arr[2]) >> 'c' You can also use negative indices to access elements from the end of the list. eg. index -1 will give you the last element of the list and index -2 will give you the second-to-last element of the list: print(arr[-1]) >> 'd' print(arr[-2]) >> 'c' https://riptutorial.com/ 425

If you try to access an index which is not present in the list, an IndexError will be raised: print arr[6] Traceback (most recent call last): File \"<stdin>\", line 1, in <module> IndexError: list index out of range Read Indexing and Slicing online: https://riptutorial.com/python/topic/289/indexing-and-slicing https://riptutorial.com/ 426

Chapter 82: Input, Subset and Output External Data Files using Pandas Introduction This section shows basic code for reading, sub-setting and writing external data files using pandas. Examples Basic Code to Import, Subset and Write External Data Files Using Pandas # Print the working directory import os print os.getcwd() # C:\\Python27\\Scripts # Set the working directory os.chdir('C:/Users/general1/Documents/simple Python files') print os.getcwd() # C:\\Users\\general1\\Documents\\simple Python files # load pandas import pandas as pd # read a csv data file named 'small_dataset.csv' containing 4 lines and 3 variables my_data = pd.read_csv(\"small_dataset.csv\") my_data # xyz #0 1 2 3 #1 4 5 6 #2 7 8 9 # 3 10 11 12 my_data.shape # number of rows and columns in data set # (4, 3) my_data.shape[0] # number of rows in data set #4 my_data.shape[1] # number of columns in data set #3 # Python uses 0-based indexing. The first row or column in a data set is located # at position 0. In R the first row or column in a data set is located # at position 1. # Select the first two rows my_data[0:2] # xyz #0 1 2 3 #1 4 5 6 https://riptutorial.com/ 427

# Select the second and third rows my_data[1:3] # xyz #1 4 5 6 #2 7 8 9 # Select the third row my_data[2:3] # xyz #2 7 8 9 # Select the first two elements of the first column my_data.iloc[0:2, 0:1] #x #0 1 #1 4 # Select the first element of the variables y and z my_data.loc[0, ['y', 'z']] #y 2 #z 3 # Select the first three elements of the variables y and z my_data.loc[0:2, ['y', 'z']] # yz #0 2 3 #1 5 6 #2 8 9 # Write the first three elements of the variables y and z # to an external file. Here index = 0 means do not write row names. my_data2 = my_data.loc[0:2, ['y', 'z']] my_data2.to_csv('my.output.csv', index = 0) Read Input, Subset and Output External Data Files using Pandas online: https://riptutorial.com/python/topic/8854/input--subset-and-output-external-data-files-using-pandas https://riptutorial.com/ 428