SQL Server Page Life Expectancy

Page Life Expectancy (PLE) can be used to measure memory pressure on a SQL Server, though it should be used with some caution.

Per Microsoft’s documentation, PLE "indicates the number of seconds a page will stay in the buffer pool without references."  In other words, it measures the average amount of time that a page will remain in the memory pool before being flushed to disk.  PLE is measured in seconds.  The lower the value, the more memory pressure on the server.

The PLE of a SQL Server instance can be checked using the following command:

SELECT *
FROM sys.dm_os_performance_counters
WHERE [object_name] LIKE ‘%Manager%’
AND [counter_name] = ‘Page life expectancy’

When examining the output of the query, note the “cntr_value” column; it contains the PLE value.

Now, some reasons to be cautious when examining the PLE:

1) Some years ago Microsoft provided some often-quoted (even today) guidance on the recommended threshold for the PLE.  That threshold was based on the hardware common when the guidance was given, and in the vast majority of cases is no longer valid today.  This blog post explains the problem very well, and provides a better way to establish what the PLE threshold should be for your server.  The recommendation given there is to use a threshold of (DataCacheSizeInGB / 4GB *300). 

As is often the case when tuning and monitoring, the best method is to monitor your server to establish the baseline PLE value during periods of normal usage (and good performance), and to pay attention when the value drops and remains below that baseline.

2) Many modern servers use Non-Uniform Memory Acess (NUMA), which means that the buffer pool may be split into multiple nodes.  On such systems, PLE may not show the entire picture of what is happening on the server.  PLE is an average across all NUMA nodes, so if only one node is under memory pressure, it may not be clearly reflected in the average value given by the PLE.  More information can be found here.

3) Memory pressure may be due to inefficient query plans, so don’t immediately assume that the server lacks enough memory to handle the workload.

Configuring SQL Server Full-Text Search

Here are step-by-step instructions for configuring tables in a SQL Server database for Full-Text search.  The following process should work for SQL Server 2005, 2008, 2008 R2, and 2012.

For more information, see Microsoft’s documentation on configuring Full-Text search:

SQL Server 2005
SQL Server 2008
SQL Server 2008 R2
SQL Server 2012

STEP 1:  Identify the tables to be full-text indexed. 

Tables to be full-text indexed must have a unique, single-column, non-nullable index.  For this example, a single table will be indexed.  Here is the DDL for the table:

CREATE TABLE dbo.Book
(  
    BookID int IDENTITY(1,1) NOT NULL CONSTRAINT PK_Book PRIMARY KEY,  
    Title nvarchar(2000) NOT NULL DEFAULT(”), 
    Authors nvarchar(2000) NOT NULL DEFAULT(”)
)
GO

– Sample data
INSERT dbo.Book (Title, Authors)
VALUES (‘A manual of the British marine Algae’,'Harvey, William H.’)
INSERT dbo.Book (Title, Authors)
VALUES (‘A manual of marine zoology for the British Isles.’,'Gosse, Philip Henry,’)
INSERT dbo.Book (Title, Authors)
VALUES (‘A history of the British sea-anemones and corals’,'Gosse, Philip Henry,’)
INSERT dbo.Book (Title, Authors)
VALUES (‘The life of North American insects’,'Jaeger, Benedict.|Preston, Henry C.’)
INSERT dbo.Book (Title, Authors)
VALUES (‘Proceedings of the California Academy of Sciences.’,'California Academy of Sciences.’)
INSERT dbo.Book (Title, Authors)
VALUES (‘The romance of natural history’,'Gosse, Philip Henry,’)
INSERT dbo.Book (Title, Authors)
VALUES (‘British conchology : or, an account of the Mollusca which now inhabit the British Isles and the surrounding seas’,'Jeffreys, John Gwyn,’)
INSERT dbo.Book (Title, Authors)
VALUES (‘Proceedings of the Entomological Society of Philadelphia.’,'Entomological Society of Philadelphia.’)
INSERT dbo.Book (Title, Authors)
VALUES (‘Observations on the Pedipalpi of North America’,'Wood, H. C.’)
INSERT dbo.Book (Title, Authors)
VALUES (‘On the origin of species, or, The causes of the phenomena of organic nature : a course of six lectures to working men’,'Huxley, Thomas Henry,’)
GO

This DDL represents a table that contains data about books.  The first column provides the unique, single-column, non-nullable index for the table, in the form of the table’s primary key.  The Title and Authors columns contain the searchable information about each book.  In a real-world scenario, the table would likely contain additional fields, both searchable and not. 

A good strategy for full-text indexing is to create a set of denormalized tables just to support full-text search, and accumulate data from the primary relational model into those tables.  This strategy separates tables used for OLTP from tables that support search functionality.  The key to making this strategy effective is deciding on an appropriate method and schedule for syncing the data between the OLTP tables and the search tables.

STEP 2: Enable the database for full-text indexing (SQL 2005 only)

In SQL Server 2008 and above, databases are always full-text enabled, so no action is needed.  In SQL 2005, full-text indexing is enabled be default.  If for some reason it is disabled, it can be enabled by executing the following command:

exec sp_fulltext_database ‘enable’

STEP 3 (OPTIONAL):  Edit the Noise Words list (SQL 2005 only)

Noise Words are very common words that, if full-text indexed, decrease the accuracy of full-text searches.  Examples are words like ‘a’, ‘the’, and ‘or’.  SQL Server 2005 ships with default lists of Noise Words for many different languages.

To view or edit the list, go to the database server and find the appropriate language file (for US English, the default file location is C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\FTData\noiseenu.txt).  If editing this file, don’t forget to make a backup!

For this example, after editing, the content of the file will be:

$
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

This represents the removal of most the contents of the file.  The table being indexed contains data about books, and many “noise” words prove to be significant for books.  For example, with the default Noise Word list, a full-text search for "origin of species" may fail to find Darwin’s best-known work (notice the noise word “of” in that title).  That is why, for this example, the content of the noise word file has been greatly reduced.

The need to modify the noise word list will vary by application; many times the file will not need to be edited.

If the noise word file is edited after the full-text catalogs have been created, they will need to be rebuilt.

STEP 4: Create a full-text catalog

A full-text catalog is a physical (SQL Server 2005) or logical collection of full-text indexes.

For SQL Server 2005, use the following command to set up a full-text catalog:

CREATE FULLTEXT CATALOG FullTextSearchCatalog IN PATH ‘D:\’ 

The path to the full-text catalog (in this case ‘D;\’) will be unique to the server on which SQL Server is installed.

Note:  As a best practice, full-text catalogs should be created on their own physical drive.

In SQL Server 2008 and after, a full-text catalog no longer has a physical implementation.  It is instead a logical concept that represents a group of full-text indexes, so no file paths are necessary.  Therefore, the command for setting up a full-text catalog in SQL Server 2008 and above is simply:

CREATE FULLTEXT CATALOG FullTextSearchCatalog

For information about other arguments for this command, look at this article on MSDN.

STEP 5: Create a full-text index

Use the CREATE FULLTEXT INDEX command to create a full-text index on a table, as shown here:

CREATE FULLTEXT INDEX ON dbo.Book
(  
    Title,  
    Authors  
)
KEY INDEX PK_Book ON FullTextSearchCatalog
GO

There are several things to notice about this command.  First, the index is being placed into the full-text catalog that was created in the previous step (FullTextSearchCatalog).  Next, the primary key of the table (PK_Book) is used to map table rows to full-text keys.  Finally, all columns to be full-text indexed (Title and Authors) are included in the index definition. 

STEP 6 (OPTIONAL): Populate the full-text indexes

This step is considered optional because the creation of the full-text index in the previous step should have automatically started the index population process.  However, this might need to be done again at a later date (if the noise words file is modified, for example).

To fully populate the full-text index, use this command:

ALTER FULLTEXT INDEX ON dbo.Book START FULL POPULATION
GO

Note the following command will accomplish the same thing:

ALTER FULLTEXT CATALOG FullTextSearchCatalog REBUILD
GO

It is also possible to update full-text indexes incrementally.  See the Microsoft’s documentation for more information about populating full-text indexes.

The status of index population can be verified by checking the PopulateStatus property of the full-text catalog.  Do this by using the FULLTEXTCATALOGPROPERTY function, as shown here:

SELECT FULLTEXTCATALOGPROPERTY(‘FullTextSearchCatalog’, ‘Populatestatus’)

A return value other than "0" indicates that index population operations are in progress.  This can be used to determine if the full-text indexes are online and available to respond to queries.  Prior to issuing a full-text query, a production system might check the full-text catalog status and issue an alternate (non-full-text) query if full-text indexes are offline.

As of SQL Server 2012, the PopulateStatus full-text catalog property is deprecated.  New development work should instead check the TableFullTextPopulateStatus property of a particular table using the OBJECTPROPERTYEX function.  Again, a return value other than “0” indicates that index population operations are in progress.

STEP 7 (OPTIONAL): Edit the default Stop List (SQL 2008 and up)

Stop Lists replace the Noise Word lists used in SQL 2005, and serve the same purpose (see STEP 3 above for more information).  Whereas Noise Word lists require manual editing of a text file on the server on which SQL Server is installed, Stop Lists can be modified via T-SQL scripts and the SQL Management Studio.

Following are the commands used to configure Noise Word lists on SQL 2008 and later.  First, create a Stop List:

CREATE FULLTEXT STOPLIST FullTextStopList FROM SYSTEM STOPLIST;

Once a stop list has been created, the sys.fulltext_stopwords table can be queried to view the contents of the Stop List.  Here is the command for viewing the English words in the Stop List:

SELECT * FROM sys.fulltext_stopwords WHERE language = ‘English’

For this example, the English-language Stop List should contain only the entries that we included in the SQL 2005 Noise Words list (STEP 3).  Because this requires removing most of the default words in the Stop List, the easiest way to produce the desired list is to drop all of the default entries and re-add only those that are needed.  Here are the commands to achieve this:

ALTER FULLTEXT STOPLIST FullTextStopList DROP ALL LANGUAGE ‘English’ ;

ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘$’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘A’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘B’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘C’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘D’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘E’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘F’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘G’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘H’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘I’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘J’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘K’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘L’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘M’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘N’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘O’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘P’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘Q’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘R’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘S’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘T’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘U’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘V’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘W’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘X’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘Y’ LANGUAGE ‘English’ ;
ALTER FULLTEXT STOPLIST FullTextStopList ADD ‘Z’ LANGUAGE ‘English’ ;

An important difference from SQL Server 2005 Noise Word lists is that Stop Lists can be applied only to selected full-text indexes.  (SQL Server 2005 Noise Word lists affect the entire server.) 

To associate the Stop List with a full-text indexed table, use the following command:

ALTER FULLTEXT INDEX ON dbo .Book SET STOPLIST FullTextStopList;

After associating a Stop List with a full-text index, the index should be repopulated (repeat STEP 6).

CONCLUSION

After completing these steps, it should be possible to submit full-text queries against the full-text indexed table.  Here is one example query that uses the new index:

SELECT  b.BookID AS ID, b.Title, b.Authors, x.[RANK]
FROM    CONTAINSTABLE(dbo.Book, (Title, Authors),
                      ‘("manual" and "British") or "Gosse"’) x
        INNER JOIN dbo.Book b ON x.[KEY] = b.BookID

If everything has been configured correctly, this query will return the following results:

ID Title                                             Authors             RANK
1  A manual of the British marine Algae              Harvey, William H.   32
2  A manual of marine zoology for the British Isles. Gosse, Philip Henry, 48
3  A history of the British sea-anemones and corals  Gosse, Philip Henry, 48
6  The romance of natural history                    Gosse, Philip Henry, 48

Hope this helps!

SQL Server – Finding the Source of Excessive I/O

I recently had to troubleshoot a poorly-performing instance of SQL Server.  The cause turned out to be a single stored procedure that was producing excessive disk I/O. 

Following is a description of the process I used to identify the problem and find its source. 

This is a summary of the Windows server and SQL Server instance that was experiencing the problem:

SQL Server 2005 Enterprise edition
Windows 2008 R2 Standard edition
23 separate user databases, 7 in use 24/7

The initial suggestion that something was wrong was slowness in multiple web sites and applications that accessed databases hosted on the server.  These sites and applications all accessed different databases, so the indication was that the problem was affecting the entire server.

1. First I opened the SQL Server Management Studio Activity Monitor, and examined the four graphs at the top of the monitor.  The graphs of “% Processor Time”, “Waiting Tasks”, and “Batch Requests/sec” were all within the expected operating ranges.  The “Database I/O” graph, on the other hand, showed that I/O was running much higher than normal, and that the high level of I/O was nearly constant.

   
   The graphs in the Sql Server Management Studio Activity Monitor show server metrics at a glance.

Note:  It is important that you are familiar with the normal operating ranges for these metrics on your server.  Without that knowledge, it is much harder to identify problems.  Every server is different, so you need to know your server.

1. Next I used this script to examine the SQL Server wait conditions.  PAGEIOLATCH_SH was higher than normal.  I now had pretty clear indications that excessive I/O was the reason the server was performing poorly.
2. The first step I took to find the cause of the excessive I/O was to check for abnormal behavior from the applications that were accessing the server.  Primary suspects were two web sites backed by the SQL Server instance, but neither were experiencing any unusual loads.  No other applications were showing signs of unusual database activity.
3. Since the applications accessing the server checked out, the next step was to look at the server itself.  I opened the Task Manager on the server and confirmed that no unexpected processes (such as backup or antivirus applications) were running.
4. Still on the server, I clicked on the “Performance” tab of the Task Manager and then clicked the “Resource Monitor…” button to open the Resource Monitor.  The Resource Monitor provides more detailed statistics for CPU, Disk, Network, And Memory resources.  By examining the Disk section, I was able to identify one particular database that was showing unusually high I/O numbers.  For that database, both “Total Bytes/sec” and “Read Bytes/sec” were high. 

   
   The Windows Resource Monitor gives in-depth details about server resource usage.
    
   I had now narrowed my search for the problem down to one particular database on the server.  In the past, I have seen out-of-date statistics cause unusual query engine behavior… such as simple queries taking much longer to run than they should and consuming far more resources than necessary.  That knowledge led me to the next step.

5. I executed UPDATE STATISTICS for the major tables in the database that was showing excessive I/O.  However, this had no affect.

   At this point, I suspected that the problem was that the query plan for one stored procedure was bad.  Why?  Simply a hunch based on past experience… I’ve encountered the problem before.  I have seen that one bad procedure query plan can have the same affect as bad statistics.   The tricky part is finding the procedure that is the problem, because procedures and queries across all databases on the server run slow when the machine is starved for resources.

   Note: There are commands like SET STATISTICS IO and SET STATISTICS TIME that help you to tune your SQL procedures and queries, but they are only useful if you know where the problem is.  They are great if you need to tune an individual query/procedure, but they don’t help you find the one procedure out of hundreds that is slowing everything down.

6. The next step was to start the SQL Server Profiler and do the following…

   a) Set up a trace that watched "RPC:Completed" events.
   b) Set a filter for “Duration” greater than or equal to 5000.  Ideally I would have set filters on “Reads” and “Writes”, but I had no idea what read/write thresholds I was looking for… so instead I chose to look at just the particularly long-running queries.
   c) Set a filter on “Database Name” to view only events in the problem database. 
   d) Collect 10 minutes of data.

   Paying close attention to the I/O statistics (Reads/Writes), I was able to identify one particular procedure that was producing around 1500000 reads each time it was executed.,  This was more than five times the number of reads produced other known "expensive" queries.  In addition, this procedure was being run over and over… as soon as one execution completed, another was started.  This was OK (that was the nature of the application that was invoking the procedure), but it was a further indication that this was the culprit… the procedure was always running.  (Recall that the initial investigation showed high I/O at a nearly constant level.)

   
   One procedure (highlighted) was producing 1500000 Reads each time it was executed.

7. Now that I had identified one procedure as the cause of the excessive I/O, the next task was to tune the procedure.  Query tuning is a topic about which many articles (and books!) have been written, so I won’t cover those details here.  Ultimately I was able to reduce the number of reads and CPU usage of the procedure by a factor of 10.  Success!
8. The last step was to confirm that I/O usage dropped to normal levels after the modifications to the stored procedure, and to continue monitoring the server to be certain that the problem had been corrected.

With this process, I was able to successfully identify and correct the cause of excessive I/O and poor performance on SQL Server.  I hope you find this helpful in troubleshooting your own performance problems.

Exporting UTF-8 Query Output from MySQL

I recently needed to export the results of a multi-table query from MySQL to a tab-delimited text file.  For someone such as myself that has only basic knowledge of MySQL, there were a couple of challenges to this seemly simple task.  The first challenge was handling the size of the query result set (millions of rows), and the second was preserving the data as UTF-8.

It took me about three attempts to get this right, so I thought I’d describe the solution here for future reference (and for anyone else who finds it useful).

STEP 1

Start by opening a Windows command prompt and navigating to the MySQL tools folder.  NOTE:  If your system’s PATH environment variable includes the path to the MySQL tools, you simply need to open a command prompt.

STEP 2

In order to preserve any Unicode characters in the query output, you must specify the correct code page before executing the query. To do this, execute the following command at the Windows command prompt:

chcp 65001

The chcp command will CHange the Code Page to code page 65001, which is the UTF-8 code page on Windows. By changing the code page, you ensure that when the output is redirected to a file, that file will be created as UTF-8.

Note that encodings are handled rather strangely on Windows; a detailed description of this can be found in the preferred answer to the question posed at http://stackoverflow.com/questions/1259084/what-encoding-code-page-is-cmd-exe-using. Note also that one of the final comments found there is “… the encoding that is used when cmd is redirecting stuff to a file… closely follows chcp”. Since we are in fact redirecting output to a file, we can safely ignore most of the encoding strangeness.

STEP 3

Now you are ready to issue the command that will export the data.  The command to use is:

mysql SCHEMANAME –q –uUSER –pPASSWORD –e “SELECT * FROM TABLENAME” > OUTPUTFILE.TSV

The following table explains the elements of the export command:

SCHEMANAME	The name of the MySQL schema/database to query.
–q	Prevents MySQL from caching the query results before outputting it.  This is important if your query returns a large amount of data and/or your system memory is limited.
-uUSER	A MySQL user that has permission to execute the query.
-pPASSWORD	The password for the MySQL user.
-e “SELECT * FROM TABLENAME”	The query to be executed.  The actual query used can be as simple as the example given here, or it can be much more complex.  By default, results are printed using the tab as the column separator.  Each row is listed on a new line.
> OUTPUTFILE.TSV	Redirects the tab-separated output of the query to a file named OUTPUTFILE.TSV.

 

For tips on producing output formats other than tab-separated, see http://giantdorks.org/alain/export-mysql-data-into-csv-or-psv-files/ for more information.

SQL Server “Could Not Produce a Query Plan” and “SET QUOTED_IDENTIFIER”

On my latest project we recently ran into a situation where execution of a particular stored procedure was producing the error “Internal Query Processor Error: The query processor could not produce a query plan.”

Investigation into this problem uncovered a number of references to foreign keys and the SET ANSI_NULLS option.  A few sources suggested that computed columns contributed to the problem.  There were also hints of bugs in certain versions of SQL Server which may or may not have been corrected in subsequent service packs.

In our case, we were up-to-date on service packs and updates.  Modifying the state of the SET ANSI_NULLS option had no affect, we were able to rule out issues with foreign keys, and the tables in question did not have any computed columns.  Further investigation suggested the SET QUOTED_IDENTIFIER option might also trigger the problem, and that turned out to be the case for us.  Setting that option to ON for the stored procedure eliminated the error.

So, based on my experience, I suggest checking the following if you encounter the “Could not produce a query plan” error when executing a stored procedure:

1. Make sure you are up-to-date with service packs and patches.
2. Include SET ANSI_NULLS ON in the procedure definition.
3. Include SET QUOTED_IDENTIFIER ON in the procedure.
4. Try eliminating foreign keys and/or computed columns and see if the error goes away.

Be aware that this problem seems to be a difficult one to pin down, and there’s no guarantee that the steps outlined here will solve your problem.

Notes from “The New Era of Work” Microsoft Launch Event

On Tuesday, November 27, 2012, Microsoft held a launch event at the Chase Park Plaza hotel in St. Louis.  This even focused on Windows 8 and Microsoft Office 2013.  I attended the Developer track, which included sessions on Windows 8, Windows Phone, and Microsoft Azure.  Following are my notes from the event.

Presenter

Jeff Brand
http://slickthought.net
http://twitter.com/jabrand

Background Info

http://aka.ms/vLabsWin8 – Windows 8 Virtual Labs
http://aka.ms/30Days – Guidance for building a Windows Store app in 30 days

Session 1: Building Modern Apps

“Modern Apps” means “Windows Store Apps”

Side-loading is allowed within corporations.   This is accomplished with a "Push" model via enterprise infrastructure (desktop management services).  Side-loaded apps are not placed into the Windows Store.

The app package that is loaded to the store or used for side-loading is a ZIP file containing specific information (including the app manifest).

It is possible to create "background tasks" that get CPU resources at all times.  This is distinct from regular apps, which are suspended (NO resources are allocated to them) when moved to the background. 

Suspended apps are placed onto the heap as-is, and receive no more resources unless "re-started".  They might eventually be terminated (without regaining focus).  As a developer, you have 5 CPU seconds to save your application’s state when it enters a suspended state.

Session 2: Designing Modern Apps

http://design.windows.com – 300 pages of design documentation

Windows Store apps should not have "chrome" (min/max buttons, navigation bars, buttons, etc)

Build a great Windows 8 Store app in 5 steps

1) Create a "best at" statement

•      One sentence
•      Be specific
•      Answer this: How is the app different from others in the same category?

2) Choose the right scenarios

• Brainstorm a bunch of scenarios for use of the app
• Remove scenarios that don’t directly line up with the "best at" statement
• Find 3-5 key scenarios

3) Pick a navigation pattern

• Hierarchical – Selecting GUI elements drills down into more details.
• Flat – Scroll right and left to view more information.  Top bar can be used for additional navigation (to show a new set of information, which also scrolls right/left).

4) Layout the content

• Use fonts consistent with recommended styles 
• Align elements to a grid.  Leave space on sides and at top/bottom to give room for native Windows 8 navigation elements.
• The "Hub page" of the app should reflect the "best at" statement and key scenarios.
• Let Visual Studio help (templates)

5) Create app bars

• Brainstorm features for the application
• Remove features that don’t related to the app’s key scenarios
• Remove features that leverage Windows 8 charms (Example: search should be implemented via the Windows Search contract, rather than natively within the app.  Implementing search via the Search contract allows users to search your app’s content from anywhere in their system at any time)
• Sort remaining features by scenario
• Group commands into related sets
• Place global commands at left of the app bar (bottom of screen)
• Place contextual commands at right of the app bar (bottom of screen)
• Keep command sets in consistent positions throughout the app

Session 3: Introducing Windows Phone 8

Three phone APIs (project types for each in VS 2012):

• .NET API for Windows Phone – Managed
• Windows Phone Runtime – Managed and Native
• WIN32 & COM – Native
  Windows Phone Runtime is subset of full WinRT, with some phone-specific extensions

Development machine requirements for created Windows Phone apps:

• 64-bit
• 4GB memory
• Windows 8 Pro
• CPU must support SLAT (more info here)
  Most Windows Phone 7 apps will run unchanged on Windows Phone 8.

Session 4: Everything Web Developers Must Known to Build Modern Apps

Most any web app that runs on IE10 will run unchanged (or minimally changed) on Windows 8.

WinRT APIs overlap in some cases with HTML5/Javascript functionality.  Two examples are Geolocation and File APIs.  Use the API that works best for you.

Host (Windows 8) checks and prevents potentially harmful HTML from being inserted into a page (via innerHTML, outerHTML).  There are code workarounds (ways of telling the OS to allow the "dangerous" action).

Windows Library for JavaScript (WinJS)

• Library for building WinRT apps using JavaScript 
• Appears to provide much of what jQuery provides
• Also provides many Windows-specific hooks, including hooks to OS events like "app starting" (including info about origination event) and "app suspending".
• Apps are not required to use WinJS
• WinJS can be used in place of or alongside other JavaScript libraries (like jQuery)

Session 5: Building Windows Store Applications With XAML

Asynchronous programming is becoming the norm, and Windows 8 apps should embrace the asynchronous model.

Asynchronous programming models on Windows

• Windows Runtime: IAsyncOperation<T>
• .NET Framework: Task<T>
• Javascript: Promises

Asynch methods…

• Are marked with the "async" modifier
• Return void or Task<T>.  Task<T> was introduced in .NET 4.0 as part of the Task Parallel Library, which is the .NET Framework preferred way of writing multi-threaded, asynchronous, and parallel code.
• Use "await" operator to yield control (apps are resumed when awaited operation completes)
• Allow composition using regular programming constructs

Example async method:

protected async override void SomeMethod()
{
     var http = new HttpClient();
     try {
          var response = await http.GetStringAsync(http://somedomain.com);
          var json = JsonObject.Parse(response);
          .
          .
          .
     }
     catch
     {
     }
}

 

No native SQL Client APIs within WinRT.  To access a database, it needs to be fronted with a service.

The open source SQLite project  has created a version of their product that runs on Windows Phone 8 and Windows Runtime.  There is no version of SQL Server that runs on those platforms.

Session 6: Building Windows 8 Apps with Windows Azure Mobile Services

Mobile Services provides data storage, notifications, authentication, diagnostics, logging, compute scaling, and storage scaling.

When you create a new mobile service, "Getting Started" code is provided for Windows 8, Windows Phone 8, and iOS.

Azure Channel URIs allocated to an application for Push Notifications will become invalid (reclaimed by Azure) if not used for 30 days.

A Misconception about OAI-PMH Metadata Formats

OAI-PMH stands for Open Archives Initiative Protocol for Metadata Harvesting. In brief, OAI-PMH is a protocol that enables publishing and harvesting of metadata about objects (such as PDFs or JPGs) in an archive.  In my experience, I have found that one aspect of this protocol is often misunderstood.

About OAI-PMH

OAI-PMH defines six different commands (called verbs). Three of these commands are used to publish 1) information about the archive itself, 2) the metadata formats in which data is published, and 3) the sets of data available in the archive. The remaining three commands are used for the actual publishing/harvesting of data: one command publishes a list of record identifiers, one publishes a list of metadata records, and one publishes a single metadata record. By supplying the appropriate arguments to the commands, it is possible to track changes to an archive since a certain date or between two dates.

Much more information about OAI-PHM can be found at http://www.openarchives.org/pmh/.

The Misconception

Having worked with OAI-PMH quite a bit the last few years, one thing that has consistently surprised me is a significant misconception about the protocol. Namely, more than once I have encountered the belief that the OAI-PMH protocol supports only the Dublin Core metadata format. This is simply not true.

It IS true that the OAI-PMH specification requires metadata to be published in Dublin Core, but it does NOT dictate that metadata be published ONLY in the Dublin Core format.

Specifically, the OAI-PMH specification states "…OAI-PMH supports items with multiple manifestations (formats) of metadata. At a minimum, repositories must be able to return records with metadata expressed in the Dublin Core format, without any qualification. Optionally, a repository may also disseminate other formats of metadata."

The Reality

If one browses the web and examines the various archives publishing with the OAI-PMH protocol, it quickly becomes clear that many archives publish in Dublin Core and also in one or more additional metadata formats. In almost every case, these additional formats provide much richer metadata than is possible with Dublin Core (formats like RDF, METS, and MODS are not uncommon). Here are some examples:

Rice University Digital Scholarship – Dublin Core, RDF, METS, two others
Gateway to Oklahoma History – Dublin Core, RDF, one other
Smithsonian Digital Repository – Dublin Core, Qualified Dublin Core, MODS
Pensoft Publishers – Dublin Core, MODS
PubMed Central – Dublin Core, two formats derived from NLM DTDs created for journal metadata exchange

This, I think, is the point… Dublin Core exists as the lowest common denominator for metadata exchange with OAI-PMH, but most archives should (and do) provide something richer.

Conclusion

The support for multiple metadata formats is the distinction that some potential users and adopters of OAI-PHM miss. I have seen archives proclaim that they support OAI-PMH, only to find that they support only the bare minimum (i.e. only Dublin Core). To them, I say "that’s nice, but given the limitations of Dublin Core, please add support for a richer metadata format". And, I have seen users dismiss OAI-PHM out of hand, complaining about its limited usefulness due to mangled Dublin Core metadata. To them, I say "look closer, and if the archives with which you are working truly only support Dublin Core, demand more".

OAI-PMH metadata formats… more than just Dublin Core.