Fixing software updates by playing with hardware

The "Epson Perfection V600 Photo" is a fine scanner, but the software often fails it, and the user.

This is true after all the most recent updates have been applied, regardless of which scanning app is used.

The error is:

"Unable to send data. Check the connection to the scanner and try again. E583-B318"

It reports this error even while visibly communicating with the scanner, with various whirring and clicking as evidence.

Not so helpfully, the error prevents scanning from taking place, after which the scanning app terminates.

With a bit of oddball tinkering, I found a "fix". I'll use the transparency scanner as an example.

If you open the lid of the scanner, and try to scan, you get this error:

"Remove the document mat from the scanner."

But this is not a fatal error. Close the scanner lid, click "ok", and try your scan.

For me (hopefully others) the scanning then works continually, until I unplug the computer from the scanner, or quit the scanning app.

I'll let the reader draw their own conclusions about the level of investment Epson makes to assure software quality ...

gmail broken with arbitrary, invisible, forced short autowrap

 ... or that's how it looked to me. 

Gmail compose took a long input line, which normally would autowrap within whatever box it was viewed, but instead created a hard, short autowrap, in the background, where I could not see it, and did not want it.

Obviously I'd accidental changed a setting. But I scoured the settings, and couldn't see an option that fit the problem. 

There's another set of settings, weirdly not referenced or linked in the main settings. These are in the compose window. And they don't apply until the next time you open the compose window.

Those settings are under the three dots to the right of the text tools, and the culprit was "plain text mode".

Now as someone who used email decades before there was email formatting, I was a little irked by the assertion implied by this setting's name.

If it was 'plain text' why not just treat the input the way it will be received? Why create an arbitrarily short autowrap of the input text, which will alway look wrong? This is because it's not previewed, that is, it's not WYSIWYG. It turns a potentially useful option into one that would only be useful for sending emails to very primitive small-screen devices, with no option to use plain text in a way that's under control of the sender. 

So, a broken UX in gmail. Which usually is more careful about its features.

Werkzeug ... and who is responsible for code stability?

Don't you love it when you haven't deployed for a few days, and you change something insignificant, and then your deployed app crashes, because of something far outside your purview?

The python Werkzeug WSGI library was just updated to 3.0. This caused Python Flask 2.2.2 web apps running on Google Cloud's App Engine to automatically update. Which, if you use one of the utilities, url_quote, you get this error:

ImportError: cannot import name 'url_quote' from 'werkzeug.urls'

So, yes, I might have caught this by updating and running it first in my local environment. But Google Cloud could have caught this too, creating a stable environment for incremental deployment.

The fix is to add this line to your requirements.txt file:

werkzeug==2.2.2

This reminds me of the whole unnecessary forced move to Flask, with its mixed bag of improvements and problems. It should be possible to run a webapp of any age (at least with configurations since 2008) in Google App Engine. Why all the unnecessary updating, crashing, and subsequent compulsory code obsolescence? What happened to backwards compatability? If it was still an observed principle, it would be easier now than ever. Why the insistence on forcing programmers to chase after the latest thing?

Error messages, Google Maps, PinElement, AdvancedMarkerElement, MapId

I often try using a new feature by adding the most local code possible, that is, right where I need it. It fails, of course. The UX for developers is very rarely a priority. When it fails, I then fix one problem at a time, step-by-step, slowly getting the feature to do its job, by working my way outwards from that first bit of code, and seeing what kinds of errors I get. I want to see whether the errors point me to the next-most general problem or, instead, obscure what's actually going on. As everyone knows, it's usually the latter.

I have a hope that someday error messages will become genuinely useful, and not just strings that we need to search for in Google, in the hope that someone like me has explained the error message, and how to make the necessary repairs.

In this case, quite a lot of time could have been saved if the Google Maps API (which is loaded) had simply caught the exceptions and told me "do (1), then (2), then (3) ..." 

Ah well. Developer advocates clearly have no power inside such giant technocracies.

The goal: I wanted a custom marker. The 'free' way to do this is by adding an 'image' name-value pair in a marker:

const bluepin = "/bluepin.png";
...
marker = new google.maps.Marker({position,map,icon: bluepin});

But there's a new set of advanced marker features. The documentation didn't match my use case (my markers are created in a listener), so I started with:

const pinBackgroundBlue = new PinElement(
    {background: "blue"});
    ...
marker = new AdvancedMarkerElement(
    {map,position: position,
     content: pinBackgroundBlue.element});

... and in the console, Javascript told me that pinElement was not found. Fair enough.

In the head element I added an import of the appropriate Google Maps libraries:

<script async defer src="https://maps.googleapis.com/maps/api/js
key=YOUR_GOOGLE_MAPS_KEY
&v=beta
&libraries=visualization,marker
&callback=initMap">
</script>

Now, in the function initMap itself, I didn't change the map declaration. 

The map is global, however.

In the listener, I added the recommended import mappings:

// Request needed libraries.
const { Map } = await google.maps.importLibrary("maps");
const { AdvancedMarkerElement, PinElement } = await google.maps.importLibrary(
"marker"
);

But since this was a listener, javascript gave me an error. The enclosing function wasn't async, so it could not await.

So I did this, which worked:

const { Map } = google.maps.importLibrary("maps");
const { AdvancedMarkerElement, PinElement } = google.maps.importLibrary("marker");


Now it didn't complain about await  / async. 

But still: "PinElement not found".

Maybe the namespace is still isolated? I really don't know how much "importLibrary" is supposed to do. So I tried:

const pinBackgroundBlue = new window.google.maps.marker.PinElement(
{background: "blue"});

Now it found PinElement!

But it said "PinElement is not a constructor".

Well ... that's just silly. Of course it is.

So, I read the documentation, watched the Google Maps videos ...

And the only thing that could be missing was something that was new to me:

A "MapId".

This is for elite maps, I suppose, since Google requires that the privilege of a generated MapId is added to your billing account. I don't know what the charge is now, or later.

But if you just want to try it, add the following to the name-value pairs in your map declaration:

mapId: 'DEMO_MAP_ID'

And now PinElement is a constructor! And your custom pin will appear on the map.

Deleting a Google Cloud Platform domain mapping under Google App Engine

Google's current documentation on this topic doesn't match the actual user interface, at least for Google App Engine custom domain mappings.

Say you want to free up a domain that you've used on a different project on GCP or GAE.

When you try to use the domain, you get an error like:

error: [domain] is already mapped to a project

To solve this, use the command line (assuming you have the developer tools and so use gcloud regularly to deploy your apps) and do the following:

gcloud auth login

This will open a browser window for you to sign in with your Google account.

Set the project in the gcloud tool to the old project where the domain is currently mapped:

gcloud config set project PROJECT_ID

Replace PROJECT_ID with the actual Project ID of the old project.

List the domain mappings for the old project:

gcloud app domain-mappings list

Locate the domains you want to remove from the list, and then run the following command to delete the domain mapping:

gcloud app domain-mappings delete DOMAIN_NAME

... replacing DOMAIN_NAME with the actual domain you want to remove.

After a bit of waiting, you'll be able use the domain again in a custom mapping.

Simplest Google App Engine Python 3.8 Flask web app with Google Identity

Tested as of this post, with Google's latest approach to sign-in ... as far as I know. With the latest python 3.8 and flask resources (as vanilla as possible ... as far as I know). 

The proper sequence (skip anything you've done already): 

1. sign up for Google App Engine

2. put down a credit card

3. download developer tools

4. create an app

5. enable all the APIs and Services you plan to use 

   (e.g., Firestore's Datastore interface, in the case below).

6. go to API Credentials on Google Cloud Console

7. add an OAuth 2.0 Client ID

8. authorize your app for javascript origins (you'll use it eventually) but most importantly for this app, authorize the redirect URIs for: 

https://app project name.appspot.com

https://app project name.appspot.com/oauth2callback

(and any domains you map to that application)

9. Everything in red on this post needs to be replaced with your own values.

This app loads index.html when not signed in, offers a sign-in link, and when the user has signed in with google, index.html includes a home page with the user info we have access to.

app.yaml

runtime: python38
app_engine_apis: true

env_variables:

CLIENT_ID: 'your client ID copied from your google cloud console'
CLIENT_SECRET: 'your client secret copied from your google cloud console'
SECRET_KEY: "the super secret key you make up"

instance_class: F1

requirements.txt

Flask==2.2.2

google-cloud-datastore==2.7.0

appengine-python-standard>=1.0.0

google-auth==2.17.1

google-auth-oauthlib==1.0.0

google-auth-httplib2==0.1.0

werkzeug==2.2.2

main.py

import os
import requests
from flask import Flask, render_template, request, redirect, url_for, session
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import Flow
from google.cloud import datastore

app = Flask(__name__)

SECRET_KEY = os.environ.get("SECRET_KEY", "the super secret key you make up")
app.secret_key = SECRET_KEY

CLIENT_ID = os.environ.get('CLIENT_ID')
CLIENT_SECRET = os.environ.get('CLIENT_SECRET')
REDIRECT_URI = "https://app project name.appspot.com/oauth2callback"

@app.route('/')
def index():

if 'userinfo' in session:

return render_template("home.html", userinfo=session['userinfo'])

return render_template("index.html")

@app.route('/login')
def login():

flow = Flow.from_client_config(

{

"web": {

"client_id": CLIENT_ID,

"client_secret": CLIENT_SECRET,

"redirect_uris": [REDIRECT_URI],

"auth_uri": "https://accounts.google.com/o/oauth2/auth",

"token_uri": "https://accounts.google.com/o/oauth2/token",

}

},

scopes=[

"https://www.googleapis.com/auth/userinfo.email",

"https://www.googleapis.com/auth/userinfo.profile",

"openid",

],

)

flow.redirect_uri = REDIRECT_URI
authorization_url, _ = flow.authorization_url(prompt="consent")
return redirect(authorization_url)

@app.route('/sign_out')
 def sign_out():
    session.pop('userinfo', None)
    return redirect(url_for('index'))

@app.route('/oauth2callback')

def oauth2callback():

flow = Flow.from_client_config(

{

"web": {

"client_id": CLIENT_ID,

"client_secret": CLIENT_SECRET,

"redirect_uris": [REDIRECT_URI],

"auth_uri": "https://accounts.google.com/o/oauth2/auth",

"token_uri": "https://accounts.google.com/o/oauth2/token",

}

},

scopes=[

"https://www.googleapis.com/auth/userinfo.email",

"https://www.googleapis.com/auth/userinfo.profile",

"openid",

],

state=request.args.get("state"),

)

flow.redirect_uri = REDIRECT_URI
flow.fetch_token(code=request.args.get("code"))
credentials = flow.credentials
userinfo = get_user_info(credentials)
session['userinfo'] = userinfo
return redirect(url_for('index'))

def get_user_info(credentials):

headers = {

"Authorization": f"Bearer {credentials.token}"

}
response = requests.get("https://www.googleapis.com/oauth2/v2/userinfo", headers=headers)
userinfo = response.json()
return userinfo

if __name__ == "__main__":

app.run(debug=True)

templates/home.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>your home page title</title>
</head>
<body>
<h1>Welcome, {{ userinfo['name'] }}</h1>
<h2>Email: {{ userinfo['email'] }}</h2>
<img src="{{ userinfo['picture'] }}" alt="Profile picture" width="100" height="100">

<a href="{{ url_for('sign_out') }}">Sign Out</a>
</body>

</html>

templates/index.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>application title</title>
<script src="https://apis.google.com/js/platform.js" async defer></script>
<meta name="google-signin-client_id" content="{{ CLIENT_ID }}">
</head>
<body>
<h1>Welcome to this web app</h1>
<a href="{{ url_for('login') }}">Sign in with Google</a>
</body>
</html>

Datastore preservation: migrating Python 2.7 to Python 3 on Google App Engine

Let's say you deploy a production web application on Google App Engine, using a Python 2.7 runtime. That's getting quite old today, and there are increasing numbers of incompatibilities you need to cope with as a result. What to do? 

It might seem daunting, and risky, to migrate to the rather different development environment of Google App Engine with the runtime of Python 3.x.

But if done properly, one worry, or risk, can be avoided. The data in datastore. If you have a great deal of it, it will be preserved during this migration, without the complications of an ETL process. If you translate your environment and webapp properly, the data doesn't go anywhere.

See for yourself.

Here are two webapp sequences, for the same simple three-tier application. 

The first is a Python 2.7 application. 

The second is a Python 3.x application. 

It's worth creating this project youself, so you can feel confident in the migration.

After this, when working on the migration of your production application, it's important to take precautions -- such as duplication of the codebase, and copying the datastore instance -- to allow rollback.

But you probably won't need it.

--------------

Sequence 1

A simple three-tier application on Google App Engine with the webapp2 framework and the python 2.7 runtime. We assume you've created your project and enabled billing, in your Google Cloud Console.

create index.html

<html>

<head>

<title><project id></title>

</head>

<body>

<div style="color:white;font-weight:bold;size:20px;">

<project id><br/><br/>

Visits {{visits}}

</div>

</body>

</html>

create <project id>.py

import cgi

import os

import webapp2

from google.appengine.ext.webapp import template

from google.appengine.api import users

from google.appengine.ext import db

class Visit(db.Model):

   visitor   = db.StringProperty()

   timestamp = db.DateTimeProperty(auto_now_add=True)

def store_visit(remote_addr, user_agent):

        Visit(visitor='{}: {}'.format(remote_addr, user_agent)).put()

def fetch_visits():

        return str(Visit.all().count())

    

class MainPage(webapp2.RequestHandler):

    def get(self):

        store_visit(self.request.remote_addr, self.request.user_agent)

        visits = fetch_visits();

template_values = { 'visits':visits}

path = os.path.join(os.path.dirname(__file__), 'index.html')

self.response.out.write(template.render(path, template_values))

app = webapp2.WSGIApplication(

          [('/', MainPage)],

            debug=True)

create app.yaml

runtime: python27

api_version: 1

threadsafe: false

handlers:

- url: /.*

 script: <project id>.app

 secure: always

 redirect_http_response_code: 301

run "python --version"

Python 2.7.4

test your app locally with:

dev_appserver.py --log_level debug .

deploy with:

gcloud app deploy --project <project id>

  

Visit the live page a few times, and find your datastore entities from Google Cloud Console.

---------------------------------------------

---------------------------------------------

  

Sequence 2

A simple three-tier application on Google App Engine with the flask framework and the python 3.8 runtime.

On your machine:

virtualenv -p python3.8.2 <project env>

source ./<project env>/bin/activate

pip install gunicorn

pip install flask

pip install google-cloud-datastore

        pip list

Create from the list results, include these two versions in a file called "requirements.txt":

Flask==2.2.2

google-cloud-datastore==2.13.2

  

[if you do this repeatedly, in different local virtual environments, you end up running "pip install -r requirements.txt" often]

create "app.yaml"

runtime: python38

  

create "main.py"

from datetime import datetime, timezone

import json

import time

import google.auth

from flask import Flask, render_template, request

from google.cloud import datastore

app = Flask(__name__)

ds_client = datastore.Client()

def store_visit(remote_addr, user_agent):

   entity = datastore.Entity(key=ds_client.key('Visit'))

   entity.update({

        'timestamp': datetime.now(timezone.utc),

        'visitor': '{}: {}'.format(remote_addr, user_agent),

   })

   ds_client.put(entity)

def fetch_visits():

        'get total visits'

        query = ds_client.query(kind="Visit")

        return len(list(query.fetch()))

@app.route('/')

def root():

    store_visit(request.remote_addr, request.user_agent)

    visits = fetch_visits()

    # NB: index.html must be in /templates

    return render_template('index.html',visits=visits)

if __name__ == '__main__':

    app.run()

 

create "templates/index.html":

<!doctype html>

<html>

<head>

<title><project id></title>

</head>

<body>

<h1><project id></h1>

<p>{{ visits }} visits</p>

</div>

</body>

</html>

set project with:

gcloud config set project <project id>

(Note this needs to be run whenever you switch to working on a different project locally, to switch the datastore your local development environment is connected to, that is, to the project's datastore.)

set credentials 

gcloud auth application-default login

(This will launch a browser -- log in, and 'allow.' Note this will connect your local environment with the remote datastore. If you want a local datastore, you need to use the datastore emulator).

run locally with:

gunicorn -b :8080 main:app

      

deploy with:

gcloud app deploy --project <project id>

      

deactivate

Check your database on google cloud console. It's still there!

Python and Javascript shared file on Google App Engine / Google Cloud Platform

The principle known as 'single source of truth' (SSoT) is quite important in computing, and is perhaps underappreciated. We need to write programs whose assumptions are consistent and easily discoverable. It's important to avoid the introduction of multiple sources for the same 'things': things that can be as simple as constants, and as complex as the definition of terminology within a theory of automation.

Sometimes, we need this to take place within a single shared file, accessible during runtime between the server and the client code, to express important editable values once.

I use local files in my development environment, to make changes to my webapp. In one important application, I don't need a makefile-like compiling stage. So, I test locally with a development server and a browser, as everyone does. But I want to have configuration files -- central, essential, important files -- which pretty much define everything critical in my application, both the universal and the contingent aspects ... and all the intermediate levels of detail needed to properly define those aspects of the work.

I call this an 'essence' or 'central description'. It is neither javascript (well, it's JSON here for technical convenience) nor python. It's just a configuration file. I've found that this compels me to maintain a quite useful habit: keep things modularized and parameterized at a higher level than code, making important ideas resuable as the application unfolds.

But how do you connect a Google App Engine Python server application (on Google Cloud Platform) to the javascript that it serves up? It seems like one should be able to use a shared JSON file easily. But that's not the case.

However, given that it's extremely important, here are the technical details of connecting them.

The shared file is shared.json.
The server-side python is called my_app.py.
The client-side javascript is my_app.js.
There's an app.yaml.
They are all in the same folder.
And I'm intentionally ignoring the html necessary to get to the javascript code, which could be embedded script, or a separate file, static, or generated ... that's all up to you. I don't know where you run your javascript, so I've just called it 'my_app.js'.

shared.json

{
 // a comment
 "name":"my_app",
 "things": {
 "one":"a"
 "two":"b"
 }
}


my_app.py

import webapp2
import io
import json

# this version of strip_comments 
# only works if the comment is defined
# by //, and // is the first non-whitespace
# on the line

def strip_comments(some_json):
 some_json = str(some_json)
 return re.sub(r'(?m)^[ \t]*//.*\n?', '', some_json)
    
class ReadAndRespond(webapp2.RequestHandler):
 def get(self):

 # get the file
 file = io.open("shared.json", "rb", buffering = 5)
 json_string = file.read()
 file.close()
 
 # strip the comments
 json_sans_comments = strip_comments(json_string)
 
 # load the JSON into a python dictionary
 dict_obj = json.loads(json_sans_comments)
 
 # get one of the values 
 my_things = dict_obj.get("things")
 
 # use it for a new JSON payload for the response
 # to the javascript request
 return_string = json.dumps(my_things, sort_keys=True, indent=4)

 self.response.write(return_string)

app = webapp2.WSGIApplication([('the_json', ReadAndRespond)])


my_app.js

 ...

 var fetch = eval( ajaxReturn('the_json') );
 var things = fetch;

 ...
 
 function ajaxReturn(xhr_url) {
    var return_string = '';
    $.ajax({
                dataType: "text",
                url : xhr_url,
                async: false,
                success : function (newContent) {
                   return_string = newContent;
                },
                error : function ()
                {
                }
    });
    return return_string;
 }
 
 ...

app.yaml

runtime: python27
api_version: 1
threadsafe: false

handlers:
- url: /shared.json
  static_files: shared.json
  application_readable: true
  secure: always
  redirect_http_response_code: 301
  
  - url: /.*
  script: my_app.app
  secure: always
  redirect_http_response_code: 301
  

Google Cloud Platform or Google App Engine 'gcloud app deploy' not updating your app? Debugger not working?

If you use python, and occasionally run your app locally, you'll notice some files that are generated: .pyc and index.yaml. If you accidentally 'gcloud app deploy' these files, especially when you have not run it locally for a while, they can break your application! Or drive you crazy!

In particular, they break the Google Cloud Console debugger (formerly the stackdriver debugger)  ... so while you're trying to figure out why your application doesn't work as expected, the debugger keeps telling you that you're setting logpoints on non-existent lines of code!

What has happened, is that Google is not checking to see whether the .py source file and the .pyc file match. It would be very nice if they did this. They could just check the timestamps. It would be easy. But they don't. 

And you keep changing your source code in trivial ways, perhaps adding some logging to see what's going on, but the live deployed application doesn't change. And the debugger doesn't work!

The fix is obvious: delete the .pyc and index.yaml files. Or put them in your .gcloudignore file.

I hope this helps someone. :-)

CHAQ: Central Handler, Action Queue

The evolution of cheq into CHAQ is pretty easy to explain. 'Events' are things that happen to you, and 'actions' are things that you do. The Central Handler in a javascript application is called by system events that we initiate in our own code, and then we let go, so the browser can do what it needs to. But before we let go, we check what actions we need to take. They are on a queue of these actions, which we've loaded within our own code. We process them, then pass control to the browser.

You can see this code, used to explain itself in a baby web app, at chaq.rocks.  

One other change -- our 'option-oriented programming', where the program itself is determined by a JSON structure, refers to the names of other JSON objects, not to functions. I found that I would otherwise not make the functions generally usable by other JSON instructions. So functions are only called by 'process'. Ultimately, for namespace sanity, our functions are injected into process, where they are called by the program described in JSON, which I call the 'essence', and which is really a not-javascript-specific declarative programming language 'structure', which can be used for any level of abstraction you like.

The point of the essence is to give people an opportunity to maintain clarity of ideas in the description of the program's activity. Coding beyond the barrier of incomprehensibility (and other desired qualities) is very common in software development. Anything that helps us to prevent that, needs to be explored further.

 

cheq: central_handler and event_queue

No matter how you look at it, even if it's hidden, every good browser-based JavaScript application must have a "central handler", and an "event queue". I call this necessity "cheq", as a mnemonic.

Why is this? Because a JavaScript program cannot hold onto (monopolize or block) the execution thread (the control flow of the browser's computational actions) and still make use of the essential services provided by the browser: rendering, user event handling, etc. We must pass control back to the browser, all the time, or nothing apparently happens.

But how do you do this, if you need your program to do "many things that are tied together", while passing control to the browser between each of these things? The answer, as I've said, is your own "event queue": a control channel under your control, which will persist while the browser is busy, say, rendering something for you. Every JavaScript programmer runs into this problem all the time: why isn't "X" appearing on the screen? Oh -- I didn't pass control back to the browser. This is especially obvious when you build animations.

If you have an event queue of your own, independent of the browser's event system, then you need a central_handler that manages that event_queue. Hence "cheq":

/* -----------------

 cheq.rocks 
 "cheq" means "central_handler event_queue".

 This needs to be at the heart of any browser-based 
 javascript application. It allows you to control your 
 program flow while cooperatively passing control to the 
 browser in order to render, handle events ... 

 The initial call from index.html looks like this:
     event_queue.push({'name':'initial_subhandler_name',
      'delay':2000});
     central_handler()

 Subsequent calls from inside the event look like:
     event_queue.push({'name':'some_subhandler_name',
      'delay':2000});
     central_handler()

 OUR EVENT QUEUE (so the browser regularly gets control):
  uses .push(x) to add to queue
  and .shift() to get the next event
*/
var event_queue = [];
var the_event = null;

// CENTRAL_HANDLER:
//  called by onload and setTimeout
function central_handler() {

    if (!(event_queue.length > 0)) {
 return;
    }
    the_event = event_queue.shift();

    // call event
    window[the_event.name]();

    // only loop until the stack is empty
    if (event_queue.length > 0) {
       setTimeout(function () {central_handler();},
    the_event.delay);
    }
}

// end of cheq.rocks
// -----------------

"Cheq" may be considered "the heart" of any JavaScript application, from one perspective. It's not necessarily the most useful idea for a program "heart", for comprehensibility. But ... maybe it is a useful central organizing principle. You never know until you try. So, I'm going to try. I'll evaluate this with my "smoothly unfolding sequence" approach, described at core memory, making good use of the Life Perception Faculty in the human brain as a means of judgment, and see if I can maintain "explanatory" reasonableness as well. My explorations in maintaining a good development structure, from this starting point, will be here: cheq.rocks

The Biology of Password Security

If you can, try to accurately recall a complete sentence that you just said, to another human being. One that wasn't very long, nor very short, nor a cliché, nor a quote. One that was grammatically correct to you. 

That sentence is more than likely unique in human history. At the very least, if you type it into Google, with quotes around it, you are unlikely to find it. Try it a few times. 

People are often under the misimpression that all human sentences must be on the internet, making up a kind of corpus of all languages. Nothing could be further from the truth. Any natural language 'corpus' is a finite set of captured sentences: they are superficial artifacts of the complex human thought that produced the sentence. Even if a corpus was, somehow, an infinite set of sentences, it wouldn't be the right infinite set, because we don't yet know what that set is.

It's not possible to get a machine to automatically generate your phrase. There is no generator for all (and only) the sentences of any human natural language, for the following very simple reason. The mechanism that produces language is in the human brain; the brain is an extremely complex biological system, and we understand some things about it, but not much. It is highly structured, but we do not know the structure, and in fact we only have a few dozen reliable hints about the structure, despite centuries of intense work by legions of linguists. Since that biological structure is a major factor for any natural language grammar, we have no worked-out grammar (syntax), in the sense of an explicit definition of an infinite set of sentences for any human language. The actual grammar is a faculty of our brain, the faculty we use to both generate sentences and evaluate whether something is grammatical. It is part of our biology, and we have no more conscious access to its detailed operation than we do about our visual system, or, for that matter, our digestive system. We must construct experiments -- testing this biological grammatical 'meter', this language faculty, in the same way we construct experiments on our visual system with optical illusions -- in order to find out things about its operation. 

This is a research initiative, and we'll all be dead before the human natural language mechanism is understood well enough to create a generator for all and only natural language sentences. 

So, it's quite safe to take any natural sentence, like this one [but not this one, of course, since it's been written down!] and use it as your password. (These are also known as passphrases).

It's also easier to remember natural sentences than all this nonsense. But it's not trivial to remember natural sentences. You need to train yourself, and learn to be sensitive to your own speaking, including speaking to yourself (we mostly use our language faculty to talk to ourselves). If you're a writer or actor you might already have practiced this facility for remembering sentences, which we often call an "ear". But anyone can do it. It's part of our genetic endowment.

If we could develop a culture of language sensitivity, we'd have far fewer problems with passwords. Those silly and unnecessary "trick" password-generators would then become a thing of the past.

One more note about infinite sets, because there's a misconception about them. There are an infinite number of infinite sets, but a very limited number of infinite sets that are human languages ... at most a finite multiple of all the people who ever lived. 

A hypothetical infinite set can be aspirational (all the sentences of English), but an actual infinite set requires a generator function. We can prove that there's an infinite subset of sentences within English ("this, and that, and this ..."), proving that the hypothetical full set is infinite. But we don't have a generator for all and only the sentences of English, or any of the other billions of languages that ever existed (assuming, again, that the upper bound is some multiple of every individual, with somewhat unique human languages of their own).

However, the joy of natural science is to discover more about the structure that is universal in all this variation ... "universal grammar" just means that part of language that is our genetic endowment. In this sense, every human language is the same. And until we understand the universal grammar, we cannot have a single complete generator for any particular natural language. Note also that we use our brains to generate what to say: language is, after all, the expression of thought. So until we understand thought, we won't be able to generate all-and-only sentences of any language. 

So, again, your real human sentence is safe from hackers. (That would have been a good one.) 

Adding a docker service to boot on Linux

If you want a docker container, or a service that starts docker containers, to startup on a system boot, the correct method is buried in the manual for the command systemctl.

The system boot technology on GNU/Linux is shifting. Many distros are no longer relying on the Unix System V approach, with the oddly named /etc/init.d directory and the rcX-type 'run control' directories for different run levels. 

The move is instead towards systemd and upstart. I'm only talking about systemd here, and the best way to install a service that depends upon docker. 

Systemd requires unit files installed, in the proper way, in its /etc/systemd/system directories. Here's an example of a unit file, which starts up the mediawiki containers service mentioned in the previous post:

[Unit]
Description=mediawiki-containers
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
WorkingDirectory=/srv/mediawiki-containers
ExecStart=/srv/mediawiki-containers/mediawiki-containers start
ExecStop=/srv/mediawiki-containers/mediawiki-containers stop

[Install]
WantedBy=multi-user.target

This file is named mediawiki-containers.service, but it shouldn't be confused with any script service files, including the one referred to (mediawiki-containers). This unit file is called a service file because it refers to other kinds of services.

Clearly it must be started after docker is up.

Now, there's a directive in here to install it to the 'multi-user target'. This is a kind of run-level definition. But it adds to the trickiness of actually installing this.

I'm installing it on Ubuntu Linux 15.10, which has systemd pre-installed.

This unit file was authored by the mediawiki folks, and they put it on your system here: /srv/mediawiki-containers/init/mediawiki-containers.service ...

So all that's needed is to tell systemd about it, using systemctl enable:

# systemctl enable /srv/mediawiki-containers/init/mediawiki-containers.service

Created symlink from
/etc/systemd/system/multi-user.target.wants/mediawiki-containers.service to
/srv/mediawiki-containers/init/mediawiki-containers.service.

Created symlink from
/etc/systemd/system/mediawiki-containers.service to
/srv/mediawiki-containers/init/mediawiki-containers.service.

#

It should then startup on the next reboot.

Modifying mediawiki-docker: adding LDAP to the mediawiki container

The 2008 release of the 2.6.24 Linux kernel opened a new world. In particular, it supported a lightweight option for encapsulating machine configuration. 

The new kernel features -- 'namespaces' and 'control groups' -- can meet our human need for a virtual machine, without the overhead of simulated machines or additional copies of an operating system.

A 'container' is a single process, which is apparently running on its own hardware. We used to do this with a 'jail' or 'sandbox', which you could access with chroot, improved by later tools and kernel patches. The term container arose with the new, better, standardized isolation, and software that easily moves these environments around.

The premier open-source management system for containers is Docker:

If you manage or build applications and services on Linux, and you want to save your work, you need Docker. Once you see what it does, you'll want it.

Let's look at using Docker for one application. 

MediaWiki is produced by WikiMedia. You use MediaWiki everyday, because it's the free software behaind wikipedia.org.

The good people at Wikimedia created an experimental install script for a complete, running mediawiki docker deployment. It installs four interacting docker containers onto one machine -- with one line. 

On a Ubuntu 15.10 server, you can type:

curl https://raw.githubusercontent.com/wikimedia/mediawiki-containers/master/mediawiki-containers | sudo bash

This is a bash script, with both install and stop/start services, within the github project wikimedia/mediawiki-containers. If you look at the file, which is called mediawiki-containers, you'll see it pulls four docker projects from the docker hub:

• tonistiigi/dnsdock
• mariadb
• wikimedia/mediawiki
• wikimedia/mediawiki-node-services

And then it builds 'images'. What are they?

An image is a docker project that can be built, so it can then be run, as a container. 

The centerpiece of a docker project is the Dockerfile. It directs the construction of the image. It downloads resources to build the image. The images can be run as containers extremely quickly, because they are already built. If you try to 'docker run' a project that has not been built, docker will download the project and build it into an image for you.

If you want to modify a docker project, you might want to pull your changes forward, to the Dockerfile, as much as possible, or, in this case, perhaps to the mediawiki-containers service file (which gets installed in /srv/mediawiki-containers on your machine). Saving them in an image tends to get lost. It's nice to destroy containers when you stop them, and then run from a fresh image.

As an example: I needed LDAP in my docker mediawiki container. One way to do this is to modify PHP on the running image (adding php5-ldap).

# docker exec -it mediawiki bash
(container id)# (make your changes)
(container id)# exit

# docker commit -m "with my modification"

But that leaves you with unnecessarily large images to keep track of. 

Why not pull the change forward to the Dockerfile? mediawiki-containers 'pulls' the docker project for mediawiki, whose github source can be found here:

https://github.com/wikimedia/mediawiki-docker

Download it (with git, or in a zip, whatever) and add to the docker file.

# cd

# wget https://github.com/wikimedia/mediawiki-docker/archive/master.zip

# apt-get update

# apt-get install unzip

# unzip master

# mv mediawiki-docker-master mediawiki

# cd mediawiki

# emacs Dockerfile

Added php5-ldap to the appropriate spot

# docker build -t greg/mediawiki .

# docker images

# cd /srv/mediawiki-containers/

# emacs mediawiki-containers

Changed run to greg/mediawiki

# service mediawiki-containers restart

# journalctl -f -u mediawiki-containers -n 0 | sed '/: Done in /q'

If you add phpinfo(); to your /srv/mediawiki-containers/data/mediawiki/LocalSettings.php file, for a moment, you'll notice that LDAP is enabled. 

You now just need to configure the LocalSettings.php file to require the mediawiki LDAP extension and set wg values for your network.

Accessibility Audit Transparency ... how you can demonstrate your WCAG 2.0 conformance effort, and encourage others

WCAG 2.0 is the current w3c standard for web accessibility. Section 508 of the ADA, the Americans with Disabilities Act, is incorporating WCAG 2.0 standards into law. If you develop a web application, and get federal money or just want to do the right thing, this is important.

There are many tools that help developers to address accessibility problems with their web interfaces. Mostly, these problems are html tags and attributes that either interfere with the operation of assistive technologies, or they are parts of features that aren't accessible through these technologies. This is fixable. A subset of HTML5 known as WAI-ARIA can make WCAG conformance much easier, and most automated audit tools suggest using aria attributes.

If you use an audit tool for development, you can make progress on conformance. But how can you show that you're making progress? 

What? Why would we care about that? 

Because web development on a live site is incremental. For a complex web application, these audit tools can point out hundreds of WCAG 2.0 problems per view or page. Depending on your team's resources, it could take months or years to address all automatically-auditable aspects of your application.

So, say that you want to, essentially, say "We care! We're actually working on it! Look!" How can you demonstrate it? Well, hypothetically, if someone on your team, who is technical, has some extra time, they can keep a spreadsheet of the fixed problems, just in case anyone asks. Or, perhaps your bug-resolution process is public, and you've logged accessibility issues as bugs: that can demonstrate your continuing efforts.

Or, you can simply record your audits as you go, with this free service, using open source software, which I'm announcing here: wcag-audit.

As you can see from this example, which is run on wcag-audit itself, I downloaded our modified Google Chrome Accessibility Audit Extension, and ran the audit, which recorded the results. Then I fixed one accessibility problem, and ran the audit again. Then I fixed another ... and I was done (it's a simple webpage).

The chrome extension has only been modified to send the summary of audit results to the wcag-audit site for recording purposes. 

You can then hand out these wcag-audit links that reference your url's, to show your progress. Or, someone else may check your progress, or your deterioration, at any time -- keeping us all honest! Let's call that 'crowd-auditing'. In any case, your work becomes a matter of public record, without any extra effort to save evidence or create reports.

This small effort emerged out of work for the University of Oregon, whose many web applications include products for K-12 schools produced by the College of Education. We want to demonstrate that we care, that we're working on accessibility, and to publicly invite everyone else to reduce their automatically-auditable accessibility problems to zero. Let's make a friendly competition out of this. Of course, those audits are not all you need to do to provide a genuinely good web experience, for people using assistive technologies, but it's an important start.

The reason we used a Chrome accessibility audit extension, versus some other form of automation: we didn't want to deal with the state of anyone's program. It's up to you, when you audit your application, to know what the user is viewing, whether they are signed in, etc.

It begs the question, to be dealt with in the future, about what to do with webapps that don't track their important states with their href. Most do not. You can keep track of the timestamp so you know which audit is yours, but the different application states will not be tracked publicly in a differentiating fashion, if wcag-audit doesn't know what the states and their values are. 

So, I would like to propose the following published states standard for webapps, SPA's, etc.: 

* hide a div with an id whose value is the href you do use, so it can be found.

* as the value of this div's html content, provide a JSON object of the safe-for-public-view names and values you need to differentiate the states of your browser-based application. 

In the next version, we will look for this hidden div. In this way, you can continue to hide your state values from the user, if you like. But you will need to start keeping track of states before you publish them in this way. So, if you do not track internal state, defined anyway you like, please start now -- in my opinion, it's a critical aspect of the future of programming, if we are to make any progress. Tracking internal states can make the inspection of program operation easier, and is a foundation for the explanation of the ideas used by particular developers to create their applications.

I should add, although it's obvious, that other kinds of audits can be carried out this way, i.e. public crowdsourced auditing, whether they are automated or not. Some already are, more or less: that's what a reputation or review is. All that's important here, is that the audits are public. 

These audits are also a kind of proof of system status -- we do this already with uptime, for example. Why not regarding other claims? Privacy, security, transparency, etc. -- all can be audited publicly, for real-time systems that commit to these values.