Accessing local resources from a Win8 app HTML web context

Posted on June 4, 2013 by Steve

Javascript Windows 8 App Store apps (AKA Modern apps AKA Metro apps) are strongly sandboxed and one of the effects of this is that access to local resources is tightly restricted to a small set of clearly de-marked locations. Furthermore, HTML in an iFrame running in a so-called “web context” has further restrictions compared to the “local context” of HTML for the native app. Media resources in local library locations such as “My Pictures” are not easily accessible from the web context but it can be achieved. I’m documenting the solution here as I worked hard to solve it and could not find any documentation at the time.

A little background

Having built a few HTML spikes of Maavis features we satisfied ourselves that it is now perfectly feasible to replace the old XUL version with a mixture of HTML5, W3C widgets and a thin native code wrapper to provide access to platform features not yet available in the browser sandbox. Going for HTML has many advantages, including portability now, good future deployment options, and a potentially large community, something that is vital for a successful open development project.

And then, the Windows 8 release landed and we decide to seriously look at it as the first platform we targeted. I won’t go into all the reasons for this but it’s interesting to note that the simplified touch interface style that is a key feature of the Maavis UI has now appeared in consumer platforms, including Windows 8.

So we spent an OpenDirective week off-site hacking and exploring how we might implement features in HTML running in a Windows App Store app native wrapper. In other words we want the core application to be HTML code that will run on other platforms without change (AKA a hybrid app). I picked access to local image files which would be pretty straight forward, or so I thought.

At this point I became painfully aware that the MSDN documentation, my preferred first port of call, was early release and the quality was simply not what I was used to from previous experience. I spent many years delving into Win32 program and system development from the days of Windows 3.1 (well actually “Windows for Work Groups”) until Vista arrived. This had accustomed me to the excellent API documentation and articles that formed part of Microsoft’s formidable developer evangelism efforts. MSDN magazine, then later a handful of CDs, and then a huge raft of DVDs, contained all tools and information you could ever want as a developer. As time passed Code Project became a “go to”community resource and it’s good to still see it is still going strong (don’t miss their Azure competition).

At this point in time, however, while APIs were clearly defined by the objects, methods and properties, critical conceptual information providing context and defining the models just wasn’t there. Sure you can often work out the map of the forest from a study of all the trees but that leaves a lot of unknown ‘there be dragons’ areas, even after much trial and error. A couple of 3rd party blog posts gave vital clues but still with many gaps, caused largely by their being written for the preview releases of Windows 8. Plus heavy weight solutions like having a local webserver exclude running on Windows RT, and the background processing possibilities are restricted in App Store Apps.

I have since found the “must read” free ebook: Programming Windows 8 Apps with HTML, CSS, and JavaScript which fills in many of the knowledge gaps (hint, at 70MB,  don’t  try to email the MOBI to your kindle, you’ll have to sideload it). Also, Microsoft have now provided many great on-line developer resources. Now the developers friend Stack Overflow has good Windows 8 discussions.

A solution

To reiterate, the problem to be solved is to display an image in HTML in a web context iFrame where the image is a file in the “My Pictures” library on the local file system.

We need a URI referencing the image for src attribute of an image tag/element and while it could theoretically be defined declaratively in the HTML or set programatically by setting the src attribute, I’ll stick to the latter as that is more much flexible in terms of possible solutions, as well see.

WinRT provides a set of URI protocols for accessing resources in various locations (the protocol is the http: part of a web URI). However, the restrictions we hit in the App web context are:

  • The file: protocol is completely blocked. This is the usual way to access local resources from HTML.
  • There is no protocol for accessing the libraries like My Pictures. ms-appx: and ms-appx-web access content in the app package, while ms-appdata: looks in the app data storage area.
  • While files in library locations can be programmatically accessed from the local context and accessed by dynamically created blob: URLs to be referenced in the HTML, such references are blocked from within the web context. This is no doubt due to cross domain restriction requirements.

But wait, the solution could be to use a local context iframe containing the image element that is nested inside the web context HTML. The nested local context will then be able to use the blob URI in an image element src attribute to access the image file. Result.

Except, that isn’t allowed either. Sigh.

But wait, the WinRT MSApp method addPublicLocalApplicationUri looks very interesting. It “exposes the provided local context URIs to arbitrary untrusted web content”. In other words we might be able to use it to allow a local context iframe to be embedded in a web context iframe.

And you know what? It works. In fact I subsequently found this solution is described at the end of a long runtime error message in Visual Studio that is easy to over look

APPHOST9605: Can’t navigate to <ms-appx:///nestedLocalContext.html> from <ms-appx-web://8f028944-29cc-4f85-8d13-c1ef987ea8f3/webContext.html> because the destination URI is in a higher security zone. You can’t navigate from a zone with lower security to a zone with higher security unless you’re navigating to a local context URI from a web context URI and you’ve registered the local context URI with the MSApp.addPublicLocalApplicationUri method.

So we can access the image file in the main app code, create a local blob URL to it and then pass that to the nested local context to be used in an image element.  As we are crossing from local domain (context), to web domain domain and back to local domain we’ll need to use the official HTML cross domain communication “postMessage” twice.  In theory it should be possible to access the inner local context directly from the outer one as they are both the same domain. However, in practice it’s not possible access an iframe’s content if it is from another domain (eg access to contentDocument etc is blocked).

To help make this all clear I’ve placed working code in an example VisualStudio project in GitHub SteveALee/Win8LocalResourceAccess.

A final note, obviously you need to tread a little carefully as addPublicLocalApplicationUri is overriding the security model which is there for good reasons. In particular watch out for untrusted code access in the web context. Being forced to restrict the cross domains interactions to postMessage is a security advantage.

But really?

Time for a quick sanity check. Is all this worth the effort? Is there really a use case or is it simply an academic exercise? Why not just remove the web context completely? Won’t other platforms have similar cross domain limitations for access local resources? Remember we want the core application to be HTML code that will run on other platforms without change. I can’t yet comment on other browser/platform combinations, but this is certainly a possible partial solution to having generic HTML code that works on various platforms. For now, I’m recording it so I can come back to it if required when we start to look at other platforms.

Posted in Apps, HTML, Win8 | 1 Comment

Screencast of Maavis and Cloud4All integration

Posted on April 23, 2013 by Steve

I finally put up a short screencast of Maavis in use. This also demonstrates how integration with Cloud4All provides automatic personalisation of settings based on a token that identifies the user. Cloud4All is an EU project that is implementing part of the GPII personalisation infrastructure.

To record I used CamStudio on Windows and the slight echo seems to be a result of turning on preview of the microphone in Windows and using the CamStudio option of recording from ‘speakers’. This allowed both my voice and Maavis synthetic speech to be mixed and recorded. You get a slight delay in hearing your voice as you record, no doubt due to double latency, but I found I got used to it.

I also made life a little harder for myself as Maavis and Cloud4All are running in a Virtual Machine (using the VMWare player). Running CamStudio in the same VM killed the response and Maavis took forever to run. So I ran CamStudio in the Host OS and recorded the VM window.

YouTube thought I was using widescreen, which I was not. This may have been because I had not full screened the VM window so the hight was reduced. However I also have my Windows TaskBar on the left whic halso reduces the widht. Who knows, I don’t think it has any practical effect.

Posted in a11y, dementia, opena11y, opensource | Leave a comment

SVG files can now be used in HTML <img>

Posted on April 4, 2013 by Steve

I missed this development which we sorely wanted a while back; all browsers now support SVG in <img>. This is great news for accessibility (symbol users) and responsive design. For example it’s easier to use the Mulberry SVG symbols set

I put up a simple test page with more details

Posted in Uncategorized | Leave a comment

Review of Laney RB1 bass combo

Posted on February 16, 2013 by Steve

Review of this great little bass combo for Amazon.co.uk.

Posted in bass, music, Uncategorized | Leave a comment

Accessing DVD menus on Android tablets

Posted on February 14, 2013 by Steve

I’ve finally figured out how to “play” DVDs on Android tablets and access the DVD menus. It’s actually quite easy but took ages to figure out from searching forums etc. My reason for wanting to do this is to access a bass guitar tutor DVD on my tablet. The tablet is placed on a music stand and wired into my amp. This configuration makes playing along much easier than with a laptop as it’s in line-of-sight and easy to access the controls.

If you only want to watch a film then you can simply rip the entire DVD or individual VOB files and transcode to a suitable format ready for transfer to or access from the Android device. Everyone and their dog knows how to do this and readily shares their wisdom about using the many free and open source tools. But this doesn’t provide access the DVD menu systems which are often designed to be for good access and navigation of the various features.

The main reason for wanting to access the menus is when using DVDs that make extensive use of them. For example, a language training course or educational title, such as my bass playing course. The alternative of ripping a zillion small files, figuring out suitable filenames from the chapters and then navigating through a linear list in a file manager is not a usable option. In addition, some DVDs have unused, hidden files left over from mastering, making it harder to figure out which files you actually want.

So the solution:

  1. Copy the DVD files to a directory/folder named “DVD” [see update below] in the root of the internal or on an external SD card. Use direct file access with a file manager to the optical drive containing the CD. Make sure the “AUDIO_TS” and “VIDEO_TS” folders are children of “DVD”. You might possibly access the files remotely across a network as long as the folder is called “DVD” but I have not tried that, opting for standalone access with the tablet..
  2. Use a video player app that supports the “DVD” folder and provides a DVD player. I found the Daroon and Wondershare players work just fine. The MX and Dice Players only provide access to individual files.
  3. If you use an external SD card then you’ll need to add the “DVD” folder’s parent to the search/scan path (see the player settings).
  4. Refresh the player, find the DVD and play it.
  5. The menus are accessed using a dedicated control “panel” that appears with the video player controls. This avoids conflict with the existing touch interface but it would be nice if you could simply touch the menu items to activate them.

I did investigate accessing ISO files from Android apps. Not only does that not appear to be readily possible, but accessing the DVD files directly has the advantage that they are smaller than an ISO which easily breaks the FAT32 4GB limit of SD cards supported by Android.

I’m not sure if the DVD folder is official, but with at least 2 players providing support it could become a de-facto standard.

Full access to DVDs from Android tablets could be useful for some users with disabilities, for example they are easier to mount on wheelchairs and less bulky However I’ve not explored the accessibility of the 2 players that I found usable. The fact that the controllers require a touch to be displayed hints at the need for some development.

[Update 15 Feb 2013] It turns out the folder name does not need to be “DVD” and it is possible to have more than one on the sd card, at least with the Wondershare player. I assume it is the “AUDIO_TS” and “VIDEO_TS” folders that are detected.  For some reason the Daroon player has no sound with these folders on my Galaxy Note.

[Update 11 Apr 2013] Norville reported in comments below a sync issue between sound and vision. Hopefully Wondershare will fix this ASAP. Norville also pointed out that W0ndershare player will also play ISO files. This makes it really flexible and AFAIK unique.

Posted in Android, Video | 20 Comments

Developing programs designed for AT users

Posted on January 30, 2013 by Steve

It can be a challenging enough to develop programs (and apps) that provide a first class user experience for users of Assistive Technology (AT) such as screen readers. There are plenty of examples of programs ‘out there’ with varying degrees of broken accessibility when used with AT. Certainly enough to demonstrate that good accessibility is not a given and requires effort. So if you want to design specifically for users of AT you can expect a good challenge indeed.

If you add the requirement to support a range of platforms then the challenge gets even more interesting . There appears to be little information ‘out there’ to guide developers in the best approaches and tools available for AT friendly cross platform solutions.  In this post I explore the underlying issues.  In a later post I’ll present some of the available solutions. While I concentrate on the technical aspects,  I do hope it is obvious that good UX design for AT users is of paramount importance.

In general there are 2 key aspects of accessibility and programs designed specifically for AT users; the accessibility of the format of  the primary information or documents and the accessibility of the User Interface (UI). Developers want to use a toolkit that enables them to ensure great interaction with both by AT users but are often not sure how to approach it.

Thoughts about developing such AT targeted programs led Prateek Tandon of ProjectPossibility to ask us this question:

is there a particularly good language / toolset / framework to build out-of-box working applications that will work on the major desktop platforms with popular screen readers such as NVDA, JAWS, and VoiceOver?

The answer to that is “it’s complicated”, especially when I generalise it to “will work with AT”  Each platform, whether desktop or mobile has development tools which will mostly create accessible application UIs, if you pay attention to detail and perform proper testing. In addition, there are several tools available for creating cross platform native  solutions,. Many of these are mature opensource projects, including those for C++, Java and Python. In addition HTML apps and widgets are fast becoming a viable alternative, though not without some complications as we’ll explore below. The requirement for 1st class AT integration adds another dimension of complexity. The ATs on each platform are different (with the notable exception of OS X and iOS, and the Win8 versions). Plus the critical Accessibility APIs that provide the glue between the apps and AT are also different.  These accessibility APIs enable AT users to explore and manipulate the application UI and be made aware of status changes or other events.

Considering screen readers, for example: on Windows there is NVDA and JAWs using MSAA, UIA and IAccessible2 APIs, on Linux we have Orca using AT/SPI, on OS X and iOS there is VoiceOver and on Android it’s EyesFree. This complexity makes it much harder to create apps that work as desired on all platforms, especially if, as a developer, you make life hard for yourself by creating your own custom controls. It’s so much better to use standard native controls provided by the OS (buttons, lists etc) as they have accessibility baked in. The next preferable choice is to use toolkits with custom controls that have well developed and tested accessibility and AT support. Developing custom controls from scratch requires you to implement all the accessibility yourself, not an easy task to get right given the variations and edge cases to be considered.

As an example of this complexity Java Swing is popular for UI development of Java programmes. However, it has a least one short-coming for AT users, namely the need to perform a fiddly manual install of an extra package for AT access on Windows (Java Access Bridge). Of course an excellent solution to this specific problem would be for the open a11y community to improve the Java Access Bridge so the problem goes away. This would also be of benefit to users of the Windows Screen reader NVDA who want to access OpenOffice. For OpenOffice users, however, another solution is looking feasible as the community are looking at porting IBM’s work on supporting the IAccessibile2 open standard from Symphony to Apache OpenOffice).

HTML applications are rapidly becoming popular with developers, and look set to become the preferred technology for creating cost effective, portable and maintainable programs or apps. A key advantage is that the browser becomes a virtual platform that is (at least in theory) the same on any platform using any browser. This requires the standards to be followed carefully by the browser developers. The W3C standards are becoming richer and active community input is helping with speed and requirements. Browsers are now multilaterally implementing the standards more accurately and Javascript performance is no longer an issue. Critically, Javascript APIs providing access to platform capabilities are being specified and implemented. Thus full featured HTML applications to match native version are now being created.

However when we consider HTML app accessibility and integration with AT there is a more complex picture. This is due to the deeper stack of software parts compared with native apps. All the parts have to work together to ensure AT can fully access and present the HTML app’s UI. The web browser itself must implement the platform Accessibility API and cleanly map HTML accessibility through it to the AT. It does this for both it’s own chrome and the HTML content being handled. The HTML apps must be carefully developed to use accessibility best practices, especially those that effect AT. In particular good markup must be used and WAI ARIA adds important extra detail necessary for accessible applications, especially for events and updates.

Again you can make life easier for yourself by sticking to the HTML defined controls or using HTML frameworks with built-in accessibility. Open frameworks such as jQuery and Dojo have had contributions from TPG, Mozilla and others who care about accessibility. Other frameworks others like fluid have had accessibility baked in. In addition, when addressing general platform portability issues, tools like Apache Cordova/PhoneGap and jQueryMobile  provide mature open source solutions with reasonable accessibility.

The accessibility of touch screen devices is adding more challenges, and also the potential for innovation in UX for AT users. This is being spearheaded by work on mobile devices (though touch is also now becoming a built in option for laptop and desktop devices as the boundaries blur eg keyboard with a tablet or Windows 8).  For example the ways blind users can access touch screens is improving through the integrated provision of solutions like VoiceOver, talkBack, kickBack and others. The solutions for others such as those with physical or cognitive advisabilities are still a way behind. This is fertile ground for research and new solutions.

We have outlined the issues involved when  creating applications designed to be used via AT and that run on several platforms. In a future post we’ll explore the options available in more detail, but I need your input. If you have any experience of developing programs with AT users in mind pop your thoughts and experience in the comments and I’ll compile them for the next post.

Posted in a11y, opena11y, opensource, web | Leave a comment

Accessibility update

Posted on June 12, 2011 by Steve

Here’s a quick update of the main open accessibility activities that are on my radar. Do drop me a line if any are of interest to you.

This has been a very busy time, especially as our new company [pre]activity takes off (more soon). I’ve not been so closely active with Mozilla and GNOME lately, but am keeping a keen eye on the interesting activity in both of these key open accessibility projects.

Electronic ciruit board wired up to a joystick controller switch

Teckla shield connected to a joystick

The big story at the moment is proposed submission of the Open Office code by Oracle and IBM to the Apache Incubator. The frantic flurry of discussion has quietened down a bit and the vote is now on. As far as open accessibility is concerned, having the OOo code in the incubator has great potential in fostering collaboration between the many interested parties. Not only does incubation develop community diversity and strength, but the use of the permissive neutral Apache licence enables engagement by all, including those interested in commercial use. This is important given the number of derived works (see the proposal link above for a chart). Thus, for example, both IBM and LibreOffice will be able to use and contribute to the core code (if they so choose). One possible benefit that’s been mentioned is the contribution of Symphony’s more mature IA2 api implementation, allowing for example, NVDA screen reader users to use OOo without sighted assistance. In addition Apache could offer a good home for the various a11y add-ons to OOo such, as those from AEGIS project.

Project Realise is nearing the end of the JISC funded period and the web site for encouraging open innovation in accessibility software is up in prototype form. As it stands it has a way to go to fully support the vision of mentoring projects to the point of open innovation through maturity in open development, but after a workshop and debriefing, many of the critical ‘people’ focussed features should be soon appearing.

The Mulberry symbol set from Straight Street has surpassed 3,000 symbols and we’re working on partner support and internationalisation with a crowdsourcing element.

Maavis simplified access to media and communications is progressing steadily, with various possible uses and collaborations being investigated. There was considerable interest generated by the Mail article with Sylvia Syms, including some commercial. One of the top priorities is to replace Skype with something that transparently does video calls without getting in the way. We’ve also be thinking about a web version. What’s interesting is that since we started the approach to consumer interfaces has rapidly embraced simpler touch interfaces, thus making it easier to implement Maavis features.

Portable accessibility of and through web apps and widgets is fast becoming very interesting indeed. Rave in Context is creating widget templates, Tekla is providing switch access via bluetooth. Both Straight Street projects and Maavis are also heading in this direction. Tablet apps are being enthusiastically snapped up at grass roots by end users and their supporters as a way to provide low cost innovative AT need solutions. This self-service is proving disruptive in the AT provision services, and raising some serious questions. Open development of web apps can have a big impact here, especially in portability and quality.

On the European front, the eAccess+ and ATIS4All thematic networks are active and spreading knowledge. Another large project to implement some parts of the GPII vision is under review by the EC and we hope will receive funding soon.

Finally, in events, I’m excited about the DevCSI & Full Measure a11y hackday that will take place on the 21/22 June. I hope we’ll see some interesting hacking and collaborations happening. We’ve certainly got a great mixture of people registered, and excellent keynote speaker in David Banes, Sandi Wassmer and Julian Harty. I’ve ordered a Tekla prototype from Komodo OpenLab and hopefully it will arrive in time for us to play with it.

Posted in opena11y, opensource | Leave a comment

Upgraded to HTML5 themes

Posted on May 12, 2011 by Steve

I just installed Richard Sheperd’s TwentyTenFive HTML5 version of the default WordPress theme and Sylvia Egger’s Accessible Five accessibility enhancing child theme.

There was a slight glitch with the name of the TwentyTenFive folder name having changed, but all seems well now after I renamed it back to ‘TwentyTenFive’.

We can just hope these both quickly make into WP core. Especially as there was a recent call for WP accessibility contributions.

Posted in opena11y, personal | Tagged , , , | Leave a comment

Safari tab key navigation disabled by default

Posted on May 3, 2011 by Steve

For some bizarre reason tab key navigation is turned off by default in the Mac Safari web browser. By this I mean hitting the tab key repeatedly does not visit each interactive element on the page.

This lack is a pain for keen keyboard users as it’s the only guaranteed way to access all links and form elements. Even if  websites were to apply the AccessKey attribute to all links and fields, browsers in general do not provide a way to discover what allocations have been made. Browsers do provide a useful default tab order, assuming the HTML markup has a reasonable element flow.

Another group of users that need tab key access  are users of simple alternative input like switches. The overlay assistive technology that such users often use translate simple gestures into tab and enter key events, thus allowing access to most of the web app doc. This is slow and painful in many cases. especially if the tab order is not well designed. However it does provide access. The common exceptions are any ‘clever’ javascript custom controls that forget to provide any keyboard access, assuming pointer only is used. This is more common than it should be and is yet another good reason to use an established accessible toolkit like jQuery UI , Dojo or Fluid Infusion for your web app.

Screen reader user also benefit from tab navigation, though screen readers provide do more useful ways to navigate the screen. For example they can list headings (assuming the document has been well structured, as screen readers can’t work miracles).

So I wonder why this is turned off by default? I use Firefox as my default browser on my Mac Book Pro, but when setting up this blog I ran Safari for a quick test of a guest user. I used the tab key to check the Accessible 2 sub theme was doing its stuff. It was broken and tab skipped straight to the search box (the first form input element), missing all the links.

There has long been an option in system prefs – > keyboard that enables tab to all fields, but even though this was on I had no joy. Eventually after a twitter discussion, I stumbled across a new (?) similar option in Safari -> Prefs -> Advanced that also need to be turned on in order to enable basic tab navigation.

Does anyone know the logic for having tab navigation turned off in Safari by default? I’m guessing someone with influence had an issue of some sort. In Windows and Linux, tab navigation is a fundamental of any UI whether it’s a dialog or an app. HTML5 allows any item to be focusable either programmatically or by the user. So having tab navigation off is a bad move.

Posted in a11y | Tagged , , , | 5 Comments

Back in the blogsphere last

Posted on May 2, 2011 by Steve

So I’ve finally set up a new blog to replace that which was lost when EduSpaces pulled the plug on the old Elgg community where I hung out. I should have moved on anyway given my focus is now open accessibility, whereas at that time I was working in education (Exeter College MIS). Unfortunately my blog export turns out to be corrupt so it will be some time before I can resurrect some of those old posts.

Since then I’ve been pretty active on Twitter as @SteveALee and whilst being socially powerful, doing so really depletes the blogging muse. Still at times there is more I want to say than can be pared down to fit the lean and mean 140.

I also work  for UK JISC funded FE/HE educational advisory service OSS Watch and I occasionally blog there as well.

So now I’m all set to go. I had intended this to be a Drupal blog, partly to gain familiarity with the excellent CMS with a view to using it for a refresh of fullmeasure.co.uk, and partly as the Drupal Accessibility community are doing great things, many of which have landed in Drupal 7.  However, my otherwise excellent and open source friendly hoster, UKFSN are still using a version of PHP that is too old for Drupal 7. So for now I knocked up a quick WordPress install. It’s such an easy out-of-the-box blog setup and it’s fresh in my mind as I’ve just used it for a couple of websites for a client.

Posted in personal | 5 Comments