<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Python - Tag - KK's Blog (fromkk)</title><link>https://fromkk.com/tags/python/</link><description>fromkk.com is my personal blog, Explore insightful posts on Python, machine learning, and other stuff. keyword: python, machine learning, programming</description><generator>Hugo 0.165.0 &amp; FixIt v0.4.5</generator><language>en</language><lastBuildDate>Thu, 02 Jul 2026 16:11:52 +0800</lastBuildDate><atom:link href="https://fromkk.com/tags/python/index.xml" rel="self" type="application/rss+xml"/><item><title>Lazy Import in Python 3.15</title><link>https://fromkk.com/posts/lazy-import-in-python-3-dot-15/</link><pubDate>Fri, 26 Jun 2026 21:49:00 +0800</pubDate><guid>https://fromkk.com/posts/lazy-import-in-python-3-dot-15/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Python 3.15 will be release in October 2026, and lazy import is my favorite feature. In general, we often import some modules in the beginning of the script, but we just use them in some functions. When performance matters, the global import will slow down the startup time. With lazy import, we can defer the import until the module is actually used.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="basic-usage"&gt;&lt;span&gt;Basic usage&lt;/span&gt;&#10; &lt;a href="#basic-usage" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;pre&gt;&lt;code&gt;lazy import json&#10;lazy from pathlib import Path&#10;&#10;print(&amp;#34;Starting up...&amp;#34;) # json and pathlib not loaded yet&#10;&#10;data = json.loads(&amp;#39;{&amp;#34;key&amp;#34;: &amp;#34;value&amp;#34;}&amp;#39;) # json loads here&#10;p = Path(&amp;#34;.&amp;#34;) # pathlib loads here&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The syntax is pertty easy, just add the &lt;code&gt;lazy&lt;/code&gt; keyword before &lt;code&gt;import&lt;/code&gt; or &lt;code&gt;from&lt;/code&gt;. After this, the module will be loaded when it&amp;rsquo;s first used. But you can&amp;rsquo;t use &lt;code&gt;lazy&lt;/code&gt; in star import or &lt;code&gt;future&lt;/code&gt; import. For example, &lt;code&gt;lazy from module import *&lt;/code&gt; and &lt;code&gt;from __future__ import xxx&lt;/code&gt; are not allowed.&lt;/p&gt;</description></item><item><title>Namespace Package in Python</title><link>https://fromkk.com/posts/namespace-package-in-python/</link><pubDate>Sun, 10 Aug 2025 18:04:00 +0800</pubDate><guid>https://fromkk.com/posts/namespace-package-in-python/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Recently, there is a &lt;a href="https://github.com/Azure/azure-cli/issues/31843#issuecomment-3125269740" target="_blank" rel="external nofollow noopener noreferrer"&gt;GitHub issue&lt;/a&gt; about namespace package in Azure CLI. I think it is a good time to write down the knowledge about namespace package.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="what-is-namespace-package"&gt;&lt;span&gt;What is Namespace Package&lt;/span&gt;&#10; &lt;a href="#what-is-namespace-package" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;If several packages share the same root folder, then the root folder is a namespace package. &lt;code&gt;subpackageA&lt;/code&gt; and &lt;code&gt;subpackageb&lt;/code&gt; can be installed separately, even in different Python path, but they can be imported as importing a single package: &lt;code&gt;import root&lt;/code&gt;.&lt;/p&gt;</description></item><item><title>Modern Pip Build Process (–use-pep517)</title><link>https://fromkk.com/posts/modern-pip-build-process-use-pep517/</link><pubDate>Sun, 24 Nov 2024 20:49:00 +0800</pubDate><guid>https://fromkk.com/posts/modern-pip-build-process-use-pep517/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Nowadays, &lt;code&gt;pyproject.toml&lt;/code&gt; becomes the standard configuration file for packaging. Compare with the old &lt;code&gt;setup.py&lt;/code&gt;, it adds two feature pep517 and pep518.&lt;/p&gt;&#10;&lt;p&gt;&lt;a href="https://peps.python.org/pep-0517/" target="_blank" rel="external nofollow noopener noreferrer"&gt;pep517&lt;/a&gt; defines two hooks: &lt;code&gt;build_wheel&lt;/code&gt; and &lt;code&gt;build_sdist&lt;/code&gt;, which is required to build the package from source. Each build backend must implement these two hooks. It makes it possible to create other build backend such as &lt;code&gt;flit&lt;/code&gt; or &lt;code&gt;poetry&lt;/code&gt;.&lt;/p&gt;&#10;&lt;pre&gt;&lt;code&gt;[build-system]&#10;# Defined by PEP 518:&#10;requires = [&amp;#34;flit&amp;#34;]&#10;# Defined by this PEP:&#10;build-backend = &amp;#34;local_backend&amp;#34;&#10;backend-path = [&amp;#34;backend&amp;#34;]&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Besides &lt;code&gt;setuptools&lt;/code&gt;, there are some other build back-end such as &lt;code&gt;hatchling&lt;/code&gt; and &lt;code&gt;flit&lt;/code&gt;. You can find the example here: &lt;a href="https://packaging.python.org/en/latest/tutorials/packaging-projects/#choosing-a-build-backend" target="_blank" rel="external nofollow noopener noreferrer"&gt;Python Packaging Uer Guide - Choosing a build backend&lt;/a&gt;&lt;/p&gt;</description></item><item><title>sys.path in Python</title><link>https://fromkk.com/posts/sys-dot-path-in-python/</link><pubDate>Sun, 11 Aug 2024 15:56:00 +0800</pubDate><guid>https://fromkk.com/posts/sys-dot-path-in-python/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Here is the process how &lt;code&gt;sys.path&lt;/code&gt; is set in Python, with some parts omitted.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="python-command-line-arguments"&gt;&lt;span&gt;Python Command Line Arguments&lt;/span&gt;&#10; &lt;a href="#python-command-line-arguments" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;By default, as initialized upon program startup, a potentially unsafe path is prepended to &lt;code&gt;sys.path&lt;/code&gt;:&lt;/p&gt;</description></item><item><title>__import__ in Python</title><link>https://fromkk.com/posts/import-in-python/</link><pubDate>Sun, 07 Apr 2024 15:58:00 +0800</pubDate><guid>https://fromkk.com/posts/import-in-python/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;It&amp;rsquo;s known that Python&amp;rsquo;s &lt;code&gt;import&lt;/code&gt; statement is implemented by &lt;code&gt;__import__&lt;/code&gt; function. In general, if we want to import a module dynamically, we can use &lt;code&gt;import_module&lt;/code&gt; function, which is a wrapper around &lt;code&gt;__import__&lt;/code&gt;.&lt;/p&gt;&#10;&lt;blockquote&gt;&#10;&lt;p&gt;The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while &lt;strong&gt;import&lt;/strong&gt;() returns the top-level package or module (e.g. pkg). &amp;ndash; &lt;a href="https://docs.python.org/3/library/importlib.html#importlib.import_module" target="_blank" rel="external nofollow noopener noreferrer"&gt;https://docs.python.org/3/library/importlib.html#importlib.import_module&lt;/a&gt;&lt;/p&gt;&#10;&lt;/blockquote&gt;&#10;&lt;p&gt;&lt;code&gt;import itertools&lt;/code&gt; and &lt;code&gt;from requests import exceptions&lt;/code&gt; can be translated to:&lt;/p&gt;</description></item><item><title>Python 3.11 Changes</title><link>https://fromkk.com/posts/python-3-dot-11-changes/</link><pubDate>Sun, 10 Dec 2023 15:24:00 +0800</pubDate><guid>https://fromkk.com/posts/python-3-dot-11-changes/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;In &lt;a href="https://github.com/Azure/azure-cli/pull/26923" target="_blank" rel="external nofollow noopener noreferrer"&gt;[Packaging] Support Python 3.11 by bebound · Pull Request #26923 · Azure/azure-cli (github.com)&lt;/a&gt; , I bumped azure-cli to use Python 3.11. We&amp;rsquo;ve bump the dependency in other PRs, I thought it should be a small PR, but in the end, a lot of changes are made.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="args-dot-getargspec"&gt;&lt;span&gt;&lt;code&gt;args.getargspec&lt;/code&gt;&lt;/span&gt;&#10; &lt;a href="#args-dot-getargspec" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;getargspec&lt;/code&gt; is dropped in 3.11. You can easily replaced it with &lt;a href="https://docs.python.org/3/library/inspect.html#inspect.getfullargspec" target="_blank" rel="external nofollow noopener noreferrer"&gt;&lt;code&gt;getfullargspec&lt;/code&gt;&lt;/a&gt; . It returns &lt;code&gt;FullArgSpec(args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations)&lt;/code&gt; instead of &lt;code&gt;ArgSpec(args, varargs, keywords, defaults)&lt;/code&gt; So &lt;code&gt;args, _, kw, _ = inspect.getargspec(fn)&lt;/code&gt; can be replaced by &lt;code&gt;args, _, kw, *_ = inspect.getfullargspec(fn)&lt;/code&gt; However, &lt;code&gt;getfullargspec&lt;/code&gt; is retained primarily for use in code that needs to maintain compatibility with the Python 2 &lt;code&gt;inspect&lt;/code&gt; module API.&lt;/p&gt;</description></item><item><title>Memory Leak in Python multiprocessing.Pool</title><link>https://fromkk.com/posts/memory-leak-in-python-multiprocessing-dot-pool/</link><pubDate>Wed, 16 Mar 2022 21:04:00 +0800</pubDate><guid>https://fromkk.com/posts/memory-leak-in-python-multiprocessing-dot-pool/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;There is a historical memory leak problem in our Django app and I fixed it recently. As time goes by, the memory usage of app keeps growing and so does the CPU usage.&lt;/p&gt;&#10;&lt;figure&gt;&lt;img src="https://fromkk.com/images/pool_before.png"&gt;&#10;&lt;/figure&gt;&#10;&#10;&lt;p&gt;After some research, I figure out the cause. Some views does not close &lt;code&gt;multiprocessing.Pool&lt;/code&gt; after using it. The problem disappears when I use &lt;code&gt;Pool&lt;/code&gt; with &lt;code&gt;with&lt;/code&gt; statement.&lt;/p&gt;&#10;&lt;figure&gt;&lt;img src="https://fromkk.com/images/pool_after.png"&gt;&#10;&lt;/figure&gt;&#10;&#10;&lt;p&gt;But I&amp;rsquo;m still interested in it and wrote some testing code. The script is run in Python 3.6.8 and produce similar result when using &lt;code&gt;multiprocessing.ThreadPool&lt;/code&gt;.&lt;/p&gt;</description></item><item><title>How to Disable Auto Strip in Charfield in Django</title><link>https://fromkk.com/posts/how-to-disable-auto-strip-in-charfield-in-django/</link><pubDate>Sun, 19 Dec 2021 21:20:00 +0800</pubDate><guid>https://fromkk.com/posts/how-to-disable-auto-strip-in-charfield-in-django/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;In Django, when edit field in admin page or post data to forms, the leading and tailing whitespace in &lt;code&gt;CharField&lt;/code&gt; and &lt;code&gt;TextField&lt;/code&gt; are removed.&lt;/p&gt;&#10;&lt;p&gt;The reason is &lt;code&gt;strip=True&lt;/code&gt; parameter in &lt;code&gt;forms.CharField&lt;/code&gt;, which is added in Djagno 1.9. You can see the discussion in &lt;a href="https://code.djangoproject.com/ticket/4960" target="_blank" rel="external nofollow noopener noreferrer"&gt;django tiket #4960&lt;/a&gt; and here is &lt;a href="https://github.com/django/django/blob/4ce59f602ed28320caf3035212cb4d1c5430da2b/django/forms/fields.py#L211" target="_blank" rel="external nofollow noopener noreferrer"&gt;source code&lt;/a&gt;. &lt;code&gt;models.CharField&lt;/code&gt; and &lt;code&gt;models.TextField&lt;/code&gt; use &lt;code&gt;formfield()&lt;/code&gt; to create form to interact with user, then both of them eventually create a &lt;code&gt;forms.CharField&lt;/code&gt;&lt;/p&gt;</description></item><item><title>Using JSONField before Django 3.1</title><link>https://fromkk.com/posts/using-jsonfield-before-django-3-dot-1/</link><pubDate>Sat, 11 Sep 2021 21:12:00 +0800</pubDate><guid>https://fromkk.com/posts/using-jsonfield-before-django-3-dot-1/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;In Django 3.1, Django support save python data into database as JSON encoded data and it is also possible to make query based on field value in JSONField. The detailed usage can be found &lt;a href="https://docs.djangoproject.com/en/3.2/topics/db/queries/#querying-jsonfield" target="_blank" rel="external nofollow noopener noreferrer"&gt;here&lt;/a&gt;. If you are using older version and want to try this feature. Though there are many packages ported this function, I recommend &lt;a href="https://github.com/laymonage/django-jsonfield-backport" target="_blank" rel="external nofollow noopener noreferrer"&gt;django-jsonfield-backport&lt;/a&gt;.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="django-jsonfield-backport"&gt;&lt;span&gt;django-jsonfield-backport&lt;/span&gt;&#10; &lt;a href="#django-jsonfield-backport" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;This package save data as JSON in database and also support JSON query. If your database meet the requirements (MySQL &amp;gt; 5.7, PG &amp;gt; 9.5, MariaDB &amp;gt; 10.2 or SQLite &amp;gt; 3.9 with &lt;a href="https://docs.djangoproject.com/en/3.1/ref/databases/#sqlite-json1" target="_blank" rel="external nofollow noopener noreferrer"&gt;JSON1&lt;/a&gt; extension), you can use JSONField like Django&amp;rsquo;s native implementation.&lt;/p&gt;</description></item><item><title>Using cibuildwheel to Create Python Wheels</title><link>https://fromkk.com/posts/using-cibuildwheel-to-create-python-wheels/</link><pubDate>Wed, 29 Jul 2020 22:53:00 +0800</pubDate><guid>https://fromkk.com/posts/using-cibuildwheel-to-create-python-wheels/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Have you ever tried to install &lt;code&gt;MySQL-python&lt;/code&gt;? It contains the C code and need to compile the code while install the package. You have to follow the steps in this articles: &lt;a href="https://ruddra.com/install-mysqlclient-macos/" target="_blank" rel="external nofollow noopener noreferrer"&gt;Install MySQL and MySQLClient(Python) in MacOS&lt;/a&gt;. Things get worse if you are using Windows.&lt;/p&gt;&#10;&lt;p&gt;Luckily, as new distribution format &lt;strong&gt;Wheel&lt;/strong&gt; has been published in &lt;a href="https://www.python.org/dev/peps/pep-0427/" target="_blank" rel="external nofollow noopener noreferrer"&gt;PEP 427&lt;/a&gt;.&lt;/p&gt;&#10;&lt;blockquote&gt;&#10;&lt;p&gt;The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment.&lt;/p&gt;</description></item><item><title>Import custom package or module in PySpark</title><link>https://fromkk.com/posts/import-custom-package-or-module-in-pyspark/</link><pubDate>Thu, 02 Apr 2020 22:24:00 +0800</pubDate><guid>https://fromkk.com/posts/import-custom-package-or-module-in-pyspark/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;First zip all of the dependencies into zip file like this. Then you can use one of the following methods to import it.&lt;/p&gt;&#10;&lt;pre&gt;&lt;code&gt;|-- kk.zip&#10;| |-- kk.py&lt;/code&gt;&lt;/pre&gt;&lt;h2 class="heading-element" id="using-py-files-in-spark-submit"&gt;&lt;span&gt;Using &amp;ndash;py-files in spark-submit&lt;/span&gt;&#10; &lt;a href="#using-py-files-in-spark-submit" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;When submit spark job, add &lt;code&gt;--py-files=kk.zip&lt;/code&gt; parameter. &lt;code&gt;kk.zip&lt;/code&gt; will be distributed with the main scrip file, and &lt;code&gt;kk.zip&lt;/code&gt; will be inserted at the beginning of &lt;code&gt;PATH&lt;/code&gt; environment variable.&lt;/p&gt;</description></item><item><title>C3 Linearization and Python MRO(Method Resolution Order)</title><link>https://fromkk.com/posts/c3-linearization-and-python-mro--method-resolution-order/</link><pubDate>Sat, 14 Mar 2020 17:37:00 +0800</pubDate><guid>https://fromkk.com/posts/c3-linearization-and-python-mro--method-resolution-order/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Python supports multiple inheritance, its class can be derived from more than one base classes. If the specified attribute or methods was not found in current class, how to decide the search sequence from superclasses? In simple scenario, we know left-to right, bottom to up. But when the inheritance hierarchy become complicated, it&amp;rsquo;s not easy to answer by intuition.&lt;/p&gt;&#10;&lt;p&gt;For instance, what&amp;rsquo;s search sequence of class M?&lt;/p&gt;&#10;&lt;pre&gt;&lt;code&gt;class X:pass&#10;class Y: pass&#10;class Z:pass&#10;class A(X,Y):pass&#10;class B(Y,Z):pass&#10;class M(B,A,Z):pass&lt;/code&gt;&lt;/pre&gt;&lt;figure class="image-size-s"&gt;&lt;img src="https://fromkk.com/images/python_mro.png"&gt;&#10;&lt;/figure&gt;&#10;&#10;&lt;p&gt;The answer is: &lt;code&gt;M, B, A, X, Y, Z, object&lt;/code&gt;&lt;/p&gt;</description></item><item><title>Torchtext snippets</title><link>https://fromkk.com/posts/torchtext-snippets/</link><pubDate>Mon, 01 Jul 2019 21:28:00 +0800</pubDate><guid>https://fromkk.com/posts/torchtext-snippets/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;h2 class="heading-element" id="load-separate-files"&gt;&lt;span&gt;Load separate files&lt;/span&gt;&#10; &lt;a href="#load-separate-files" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;data.Field&lt;/code&gt; parameters is &lt;a href="https://torchtext.readthedocs.io/en/latest/data.html#torchtext.data.Field" target="_blank" rel="external nofollow noopener noreferrer"&gt;here&lt;/a&gt;.&lt;/p&gt;</description></item><item><title>Circular Import in Python</title><link>https://fromkk.com/posts/circular-import-in-python/</link><pubDate>Sun, 10 Mar 2019 10:59:00 +0800</pubDate><guid>https://fromkk.com/posts/circular-import-in-python/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;Recently, I found a really good example code for Python circular import, and I&amp;rsquo;d like to record it here.&lt;/p&gt;&#10;&lt;p&gt;Here is the code:&lt;/p&gt;&#10;&lt;pre&gt;&lt;code&gt;# X.py&#10;def X1():&#10; return &amp;#34;x1&amp;#34;&#10;&#10;from Y import Y2&#10;&#10;def X2():&#10; return &amp;#34;x2&amp;#34;&lt;/code&gt;&lt;/pre&gt;&lt;pre&gt;&lt;code&gt;# Y.py&#10;def Y1():&#10; return &amp;#34;y1&amp;#34;&#10;&#10;from X import X1&#10;&#10;def Y2():&#10; return &amp;#34;y2&amp;#34;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Guess what will happen if you run &lt;code&gt;python X.py&lt;/code&gt; and &lt;code&gt;python Y.py&lt;/code&gt;?&lt;/p&gt;&#10;&lt;p&gt;Here is the answer, the first one outputs this:&lt;/p&gt;</description></item><item><title>Python Dictionary Implementation</title><link>https://fromkk.com/posts/python-dictionary-implementation/</link><pubDate>Sun, 17 Feb 2019 21:48:00 +0800</pubDate><guid>https://fromkk.com/posts/python-dictionary-implementation/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;h2 class="heading-element" id="overview"&gt;&lt;span&gt;Overview&lt;/span&gt;&#10; &lt;a href="#overview" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;ol&gt;&#10;&lt;li&gt;CPython allocation memory to save dictionary, the initial table size is 8, entries are saved as &lt;code&gt;&amp;lt;hash,key,value&amp;gt;&lt;/code&gt; in each slot(The slot content changed after Python 3.6).&lt;/li&gt;&#10;&lt;li&gt;When a new key is added, python use &lt;code&gt;i = hash(key) &amp;amp; mask&lt;/code&gt; where &lt;code&gt;mask=table_size-1&lt;/code&gt; to calculate which slot it should be placed. If the slot is occupied, CPython using a probing algorithm to find the empty slot to store new item.&lt;/li&gt;&#10;&lt;li&gt;When 2/3 of the table is full, the table will be resized.&lt;/li&gt;&#10;&lt;li&gt;When getting item from dictionary, both &lt;code&gt;hash&lt;/code&gt; and &lt;code&gt;key&lt;/code&gt; must be equal.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;h2 class="heading-element" id="resizing"&gt;&lt;span&gt;Resizing&lt;/span&gt;&#10; &lt;a href="#resizing" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;When elements size is below 50000, the table size will increase by a factor of 4 based on used slots. Otherwise, it will increase by a factor of 2. The dictionary size is always &lt;span class="katex"&gt;&lt;span class="katex-mathml"&gt;&lt;math xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;annotation encoding="application/x-tex"&gt;2^{n}&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;span class="katex-html" aria-hidden="true"&gt;&lt;span class="base"&gt;&lt;span class="strut" style="height:0.6644em;"&gt;&lt;/span&gt;&lt;span class="mord"&gt;&lt;span class="mord"&gt;2&lt;/span&gt;&lt;span class="msupsub"&gt;&lt;span class="vlist-t"&gt;&lt;span class="vlist-r"&gt;&lt;span class="vlist" style="height:0.6644em;"&gt;&lt;span style="top:-3.063em;margin-right:0.05em;"&gt;&lt;span class="pstrut" style="height:2.7em;"&gt;&lt;/span&gt;&lt;span class="sizing reset-size6 size3 mtight"&gt;&lt;span class="mord mtight"&gt;&lt;span class="mord mathnormal mtight"&gt;n&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;.&lt;/p&gt;</description></item><item><title>CSRF in Django</title><link>https://fromkk.com/posts/csrf-in-django/</link><pubDate>Wed, 07 Nov 2018 13:58:00 +0800</pubDate><guid>https://fromkk.com/posts/csrf-in-django/</guid><category domain="https://fromkk.com/categories/programming/">Programming</category><description>&lt;p&gt;CSRF(Cross-site request forgery) is a way to generate fake user request to target website. For example, on a malicious website A, there is a button, click it will send request to &lt;a href="https://www.B.com/logout" target="_blank" rel="external nofollow noopener noreferrer"&gt;www.B.com/logout&lt;/a&gt;. When the user click this button, he will logout from website B unconsciously. Logout is not a big problem, but malicious website can generate more dangerous request like money transfer.&lt;/p&gt;&#10;&lt;h2 class="heading-element" id="django-csrf-protection"&gt;&lt;span&gt;Django CSRF protection&lt;/span&gt;&#10; &lt;a href="#django-csrf-protection" class="heading-mark"&gt;&#10; &lt;svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"&gt;&lt;path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"&gt;&lt;/path&gt;&lt;/svg&gt;&#10; &lt;/a&gt;&#10;&lt;/h2&gt;&lt;p&gt;Each web framework has different approach to do CSRF protection. In Django, the validation process is below:&lt;/p&gt;</description></item></channel></rss>