Difference between revisions of "Software Development"

From The Thinkulum
Jump to navigation Jump to search
(Added the article.)
 
(Moved the code examples and bibliography to a separate Sources article.)
(17 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Introduction ==
== Introduction ==


For most of my coding life, I have been the only user of my programs. I wrote them to aid my work or personal life, and only rarely did anyone else even see my code or watch it run. This wasn't on purpose. It just turned out that way. In my current job I do belong to a small development team, but our programs are still only for in-house use, so the only needs we have to consider are our own.
=== Purpose ===


This situation is starting to change. My programming projects are becoming more relevant to the outside world, and I want to share some of them as open source. This means I'm needing to reshape some of my programming practices and learn some areas of software development I've had the luxury of ignoring all these years.
This project is a growing set of notes on various aspects of software development. Its purpose is to define a set of standard operating procedures for my programming projects.


I haven't completely ignored these areas, though. I pick up on them here and there as I read about how other people code or see them in the software I use. Knowing that I'll probably need to know about them someday, I've kept a list. And now the time has come for me to fill in the details.
=== Background ===


I'm filling in the details here in this guide for two reasons. First, as I learn, writing helps me clarify my thoughts and keep them flowing. And second, posting my writing might help other people. So if you're like me and you've programmed mainly for yourself--what I call private coding--and you're starting to share your work with others--public coding--then maybe what I've learned so far can grease some of your own planning.
For most of my coding life, I have been the only user of my programs. I wrote them to aid my work or personal life, and only rarely did anyone else even see my code or watch it run. This wasn't on purpose. It just turned out that way. In my current job I do belong to a small development team, but our programs are still only for in-house use, so the only needs we have to consider are our own.


This project consists of my reading other developers' advice, examining their code, deciding what to adopt and how to adapt it to my needs and preferences, and creating templates, procedures, reference materials, and anything else that might help me code well for a public user base.
This situation is starting to change. My programming projects are becoming more relevant to the outside world, and I want to share some of them as open source. This means I'm needing to reshape some of my programming practices and learn some areas of software development I've had the luxury of ignoring all these years.


To give you an idea of my background, I've worked for the past 15 or so years in the publishing industry doing mostly text processing and some web development. The languages I'm most familiar with are Perl, Python, XSLT, and some JavaScript, with a smattering of PHP and VBA. I've spent almost all my time in Windows. I hope to branch out into at least Linux in the future, if only to expand my mind.
I haven't completely ignored these areas, though. I pick up on them here and there as I read about how other people code or as I see them in the software I use. Knowing that I'll probably need to know about them someday, I've kept a list. And now the time has come for me to fill in the details.


This guide will be oriented toward Python for now, but I imagine a lot of its suggestions apply to other languages too.
I'm filling in the details here in this project for two reasons. First, as I learn, writing helps me clarify my thoughts and keep them flowing. And second, posting my writing might help other people. So if you're like me and you've programmed mainly for yourself--what I call private coding--and you're starting to share your work with others--public coding--then maybe what I've learned so far can grease some of your own planning.


Since this guide is a place for me to learn, I welcome feedback, and I'm always open to arguments that my choices could be improved or suggestions for things I've missed.
=== Method ===


This is an early iteration of the guide, so it's incomplete, and a lot of it is only sketched out.
This project consists of my reading other developers' advice, examining their code, deciding what to adopt and how to adapt it to my needs and preferences, and creating templates, procedures, reference materials, and anything else that might help me code well for a public user base.


== General coding guides ==
To show how I'm implementing these ideas in my own code, I'll refer to [https://github.com/thinkulum my GitHub repositories] throughout the project, mainly my templates for [https://github.com/audreyr/cookiecutter cookiecutter], a Python app that creates project skeletons for any language, and my snippets for [https://www.sublimetext.com/ Sublime Text 2], the text editor I use. Snippets are templates Sublime inserts wherever you type the corresponding trigger strings.


These are resources with advice on many aspects of writing good code. The ones I haven't read are on my to-read list. I'll probably be adding others.
=== Limitations ===


* [https://en.wikipedia.org/wiki/Code_Complete Code Complete - Wikipedia]
To give you an idea of my background, I've worked for the past 15 or so years in the publishing industry doing mostly text processing and some web development. The languages I'm most familiar with are Perl, Python, XSLT, and some JavaScript, with a smattering of PHP and VBA. I've spent almost all my time in Windows. I hope to branch out into at least Linux in the future, if only to expand my mind.
* [https://en.wikipedia.org/wiki/The_Art_of_Unix_Programming The Art of Unix Programming - Wikipedia]
* [https://en.wikipedia.org/wiki/Perl_Best_Practices Perl Best Practices - Wikipedia]
 
Another resource that overlaps with this guide but covers more on the project management end of OSS is [http://producingoss.com/ Producing Open Source Software].
 
== Code examples ==
 
To show how I'm implementing these ideas in my own code, I'll refer to [https://github.com/thinkulum my GitHub repositories] throughout the guide, mainly my Code Console project, which lets me create and manage projects using templates.
 
Here are projects that show up in lists of high quality code, so I'll examine some of them and include any relevant discoveries in the guide. All the projects I'm listing for now are in Python.
 
* [https://launchpad.net/bzr Bazaar]
* [https://github.com/boto/boto3 boto3]
* [https://github.com/django/django Django]
* [https://github.com/pallets/flask Flask]
* [https://github.com/gevent/gevent gevent]
* [https://www.mercurial-scm.org/downloads Mercurial]
* [https://github.com/nltk/nltk NLTK]
* [https://github.com/Pylons/pyramid Pyramid]
* [https://github.com/reddit/reddit reddit]
* [https://github.com/kennethreitz/requests Requests]
* [https://github.com/zzzeek/sqlalchemy SQLAlchemy]
* [https://github.com/kennethreitz/tablib Tablib]
* [https://github.com/tornadoweb/tornado Tornado]
* [http://trac.edgewall.org/browser/trunk/trac trac]
* [https://github.com/twisted/twisted Twisted]
* [https://github.com/rg3/youtube-dl youtube-dl]
 
In addition to these, I keep a mental list of software with features I might want to emulate, even if I don't plan to study their code very thoroughly:
 
* [https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Source_Code Firefox] - automatic updating; possibly its add-on architecture, if it ever settles down
* [https://wordpress.org/download/source/ WordPress] - plugin hooks
 
== Software development methodology ==
 
I'll start with some general principles that direct my thinking about software development in general. These are the ideas behind agile development practices. [https://www.agilealliance.org/agile101/ Agile software development] is a paradigm that was created to answer waterfall development, the approach of planning all the features of a piece of software in advance, an effort that often ends up wasted because the customer's needs change in the middle of the project, so the software has to be redesigned.
 
So agile development was intended to prevent overplanning. But using any development methodology will help prevent the opposite problem, which is a complete ''lack'' of structure and discipline. This results in a code organization scheme called a [http://www.laputan.org/mud/ big ball of mud]. I don't think my programs were complete mud, but the fact that I was the only one seeing or using my code let me get by with a lot of slacking. Slacking does keep you from working too hard, but it fails to prevent other kinds of problems. Agile software development seems like a good middle ground.
 
Agile development encompasses a lot of principles and practices, and it has a lot of varieties, but for me at this point, its most important practices are incremental development, test-driven development, and refactoring.
 
=== Incremental development ===
 
[https://en.wikipedia.org/wiki/Iterative_and_incremental_development Incremental development] is the approach of adding a few features per release rather than developing the whole program at once. It lets users begin using the software earlier, and it lets the feature roadmap change without wasting much development effort on the original design. It's different from iterative development, but they normally go together, so I'm use incremental as shorthand for both.
 
If you're a developer working on your own, the habit of small, frequent releases also can add accountability to the development process. It'll at least hold you accountable to work on the project, since other people will notice when the releases stop coming. It might even encourage you to code each feature well. A short release cycle does mean there's less time to get the code right, but there's also less time to procrastinate on getting it to work, and if you implement good programming practices, you'll waste less of that time debugging.
 
Small, frequent releases can also help the developer avoid procrastination, since the tasks for a small release feel more achievable, similar to writing a [http://volokh.com/posts/1182282037.shtml zeroth draft] of a manuscript. There's little concern for polish, so a zeroth draft has fewer requirements to worry about, and therefore it's more likely to get written.
 
A related practice is to deliberately trim your feature list for each release, captured in the [http://wiki.c2.com/?ExtremeProgramming Extreme Programming] practice [http://wiki.c2.com/?YouArentGonnaNeedIt You Aren't Gonna Need It].
 
One practice that's helping me think in terms of incremental development is using [http://scottchacon.com/2011/08/31/github-flow.html GitHub flow] as my version control workflow. In GitHub flow the master branch is reserved for deployable code, so anything I'm actively developing goes into a topically named branch off of master. Once the code related to that branch is ready for deployment, I merge that branch back into master. This procedure gets me to plan in terms of small clusters of features. I'll talk more about version control in the next main section.
 
=== Test-driven development ===
 
* [https://martinfowler.com/bliki/TestDrivenDevelopment.html TestDrivenDevelopment - Martin Fowler]
* [http://agiledata.org/essays/tdd.html Introduction to Test Driven Development (TDD) - Agile Data]
* [https://www.facebook.com/notes/kent-beck/rip-tdd/750840194948847/ RIP TDD - Kent Beck - Facebook]
 
To get myself into the habit of writing tests first, I'm including it as a subtask for each function I note in my task manager (I use [https://nirvanahq.com/ Nirvana], an app designed with [https://en.wikipedia.org/wiki/Getting_Things_Done GTD] in mind). I have a template for these subtasks, which I copy into the description of the tasks. At the moment the template looks like this:
 
<pre>
- Document
- Test
- Code
- Commit
</pre>
 
Really I'll move back and forth between documenting, testing, and coding, but I'll check them off when I've completed the first instance of each of those subtasks for that function.
 
=== Refactoring ===
 
* [https://martinfowler.com/books/refactoring.html Refactoring - Martin Fowler]
 
I don't have a specific refactoring procedure in place. I just do it when I notice the need. I'm hoping to study Fowler's book and learn to think in terms of the structures that refactoring creates. That way I can create them as I code and reduce the need to refactor.
 
== Version control ==
 
* [http://guides.beanstalkapp.com/version-control/intro-to-version-control.html An introduction to version control - Beanstalk Guides]
 
I'm hosting my projects on GitHub (see the Distribution section), which means I'm using Git as my version control system. (Before that I was using Subversion via TortoiseSVN, which I still use for some non-GitHub projects.) I learned how to use Git from ''Git Essentials'' by Ferdinando Santacroce, and I try to follow its advice. For a reference I use ''Git Pocket Guide'' by Richard Silverman. Rather than typing all the Git commands myself, I'm lazy and use the GUI tool [https://www.gitkraken.com/ GitKraken]. As I said above, I use the GitHub flow workflow to organize my commits. GitHub has as [https://guides.github.com/introduction/flow/ guide] for it.
 
== Coding style ==
 
* [https://www.python.org/dev/peps/pep-0008/ PEP 8 -- Style Guide for Python Code - Python.org]
 
I've picked up my coding style from various sources, including ''Perl Best Practices'', but PEP 8 is a good starting point that's specifically geared toward Python.
 
== Project structure ==
 
* [http://docs.python-guide.org/en/latest/writing/structure/ Structuring Your Project - The Hitchhiker's Guide to Python]
* [https://learnpythonthehardway.org/book/ex46.html Exercise 46: A Project Skeleton - Learn Python the Hard Way]
 
I make a directory on my local machine named whatever I'm using for the GitHub URL (e.g., math-student-sim), and I create this structure inside it:
 
<pre>
docs/
<package>/
tests/
app.py
LICENSE.md
MANIFEST.in
README.md
setup.py
</pre>
 
My package name is usually the project name without the hyphens (e.g., mathstudentsim).
 
I'll fill in more of the details in later sections.
 
== Distribution ==
 
To distribute your code you need somewhere to host it, and I've chosen GitHub, since it's the one I hear the most about.
 
* [http://kbroman.org/github_tutorial/ git/github guide - Karl Broman]
 
== Installation ==
 
I haven't completely wrapped my mind around Python code distribution and installation, but for now I'm relying on people downloading the source from GitHub and running the setup script. Here are some more details on some of the issues that came up for me in the project structure guides above.
 
* [https://docs.python.org/3/distutils/setupscript.html Writing the Setup Script - Distributing Python Modules (Legacy version)]
* [https://caremad.io/posts/2013/07/setup-vs-requirement/ setup.py vs requirements.txt - caremad]
* [http://stackoverflow.com/questions/7923509/how-to-include-docs-directory-in-python-distribution How to include docs directory in python distribution - Stack Overflow]
 
== Metadata ==
 
Users of your project will need certain information about it, and you'll need to keep this information somewhere. For Python modules this is in the setup script.
 
* [https://www.python.org/dev/peps/pep-0345/ PEP 345 -- Metadata for Python Software Packages 1.2 - Python.org]
 
=== Version numbers ===
 
* [https://www.python.org/dev/peps/pep-0396/ PEP 396 -- Module Version Numbers - Python.org]
* [http://semver.org/ Semantic Versioning]
* [https://git-scm.com/book/en/v2/Git-Basics-Tagging Git Basics - Tagging - Git]
* [https://help.github.com/articles/creating-releases/ Creating Releases - GitHub Help]
 
I'm just following the guidelines in those articles.
 
=== License ===
 
* [http://choosealicense.com/ Choose an open source license]


I've picked MIT as my default to give people freedom to use the code however they want and because I don't expect to have patents to license.
This project will be oriented toward Python for now, but I imagine a lot of its suggestions apply to other languages too. At this point it also centers around desktop command-line programs, but I'm hoping to expand it into other interfaces and environments.


== Tests ==
=== Feedback ===


Based on the advice in ''Learn Python the Hard Way'', I've chosen nose as a test management tool, but since the [http://nose.readthedocs.io/en/latest/ future of nose] is uncertain, I'm using [https://github.com/nose-devs/nose2 nose2].
Since this project is a place for me to learn, I welcome feedback, and I'm always open to arguments that my choices could be improved or suggestions for things I've missed.


== User interface ==
== Topics ==


=== Command line ===
As a way to organize my notes and make sure I don't miss anything important, I'll use the structure of IEEE's ''Guide to the Software Engineering Body of Knowledge'' (SWEBOK). It's a standard that overviews the entire field and forms the basis for creating things like curricula and certifications. You can download it for free [https://www.computer.org/web/swebok/index as a PDF] or read it online [https://www.iso.org/obp/ui/#iso:std:iso-iec:tr:19759:ed-2:v2:en as HTML].


To make my projects usable quickly, I'm starting most of them with a command line interface.
The items under the first level link to my notes on those topics.


* [https://wiki.python.org/moin/CmdModule CmdModule - The Python Wiki]
* Software requirements
* [https://docs.python.org/3/library/cmd.html cmd - The Python Standard Library]
* Software design
* Software construction
** [[/Code Style/]]
** [[/Project Structure/]]
** [[/Installation/]]
** [[/Metadata/]]
** [[/Command Line Interface/]]
** [[/Documentation/]]
*** [[/Comments/]]
*** [[/Literate Programming/]]
*** [[/Docstrings/]]
*** [[/Project Documentation/]]
*** [[/READMEs/]]
*** [[/Command Line Help/]]
** [[/Application Configuration/]]
** [[/Logging/]]
** [[/Test-First Programming/]]
* Software testing
** [[/Testing/]]
* Software maintenance
** [[/Refactoring/]]
* Software configuration management
** [[/Version Control/]]
** [[/Version Numbers/]]
** [[/Distribution/]]
* Software engineering management
* Software engineering process
** [[/Software Development Methodology/]]
*** [[/Iterative and Incremental Development/]]
* Software engineering models and methods
* Software quality
* Software engineering professional practice
** [[/License/]]
* Software engineering economics
* Computing foundations
* Mathematical foundations
* Engineering foundations


I'm placing the commands in cli.py within the package.
== [[/Sources/]] ==


[More sections to come]
See the article linked in the heading for a list of code examples and other sources I'm drawing from.


<disqus/>
<disqus/>


[[Category:Programming]]
[[Category:Programming]]
[[Category:Essay]]
[[Category:Essays]]
[[Category:Developing]]
[[Category:Developing]]

Revision as of 22:44, 23 February 2019

Introduction

Purpose

This project is a growing set of notes on various aspects of software development. Its purpose is to define a set of standard operating procedures for my programming projects.

Background

For most of my coding life, I have been the only user of my programs. I wrote them to aid my work or personal life, and only rarely did anyone else even see my code or watch it run. This wasn't on purpose. It just turned out that way. In my current job I do belong to a small development team, but our programs are still only for in-house use, so the only needs we have to consider are our own.

This situation is starting to change. My programming projects are becoming more relevant to the outside world, and I want to share some of them as open source. This means I'm needing to reshape some of my programming practices and learn some areas of software development I've had the luxury of ignoring all these years.

I haven't completely ignored these areas, though. I pick up on them here and there as I read about how other people code or as I see them in the software I use. Knowing that I'll probably need to know about them someday, I've kept a list. And now the time has come for me to fill in the details.

I'm filling in the details here in this project for two reasons. First, as I learn, writing helps me clarify my thoughts and keep them flowing. And second, posting my writing might help other people. So if you're like me and you've programmed mainly for yourself--what I call private coding--and you're starting to share your work with others--public coding--then maybe what I've learned so far can grease some of your own planning.

Method

This project consists of my reading other developers' advice, examining their code, deciding what to adopt and how to adapt it to my needs and preferences, and creating templates, procedures, reference materials, and anything else that might help me code well for a public user base.

To show how I'm implementing these ideas in my own code, I'll refer to my GitHub repositories throughout the project, mainly my templates for cookiecutter, a Python app that creates project skeletons for any language, and my snippets for Sublime Text 2, the text editor I use. Snippets are templates Sublime inserts wherever you type the corresponding trigger strings.

Limitations

To give you an idea of my background, I've worked for the past 15 or so years in the publishing industry doing mostly text processing and some web development. The languages I'm most familiar with are Perl, Python, XSLT, and some JavaScript, with a smattering of PHP and VBA. I've spent almost all my time in Windows. I hope to branch out into at least Linux in the future, if only to expand my mind.

This project will be oriented toward Python for now, but I imagine a lot of its suggestions apply to other languages too. At this point it also centers around desktop command-line programs, but I'm hoping to expand it into other interfaces and environments.

Feedback

Since this project is a place for me to learn, I welcome feedback, and I'm always open to arguments that my choices could be improved or suggestions for things I've missed.

Topics

As a way to organize my notes and make sure I don't miss anything important, I'll use the structure of IEEE's Guide to the Software Engineering Body of Knowledge (SWEBOK). It's a standard that overviews the entire field and forms the basis for creating things like curricula and certifications. You can download it for free as a PDF or read it online as HTML.

The items under the first level link to my notes on those topics.

Sources

See the article linked in the heading for a list of code examples and other sources I'm drawing from.

<disqus/>

Retrieved from "https://www.thinkulum.net/w/index.php?title=Software_Development&oldid=403"