• EVAN   2014/7/15 12:55:00
  • Lucene.net学习-如何将文档文件(Word, Pdf等) 中的文本提取出来
  • 关键字: Lucene.net 文档索引
  • 经常有人问我怎么将类似word,pdf这样的文档转换为文本然后索引,.net 这方面的解决方案不是很多,为了方便大家,我花了一天时间自己做了一个。

    Java 版本的 lucence 提供了一个 tika 的工具用于将 word, excel, pdf 等文档转换为文本,然后进行索引。但这个工具没有 .net 版本,要在 .net 下用,需要用 IKVM.net,很麻烦。而且这个工具实际上底层是调用 POI 和 PDFParse 来转换的。从网上搜索到的信息看,POI 对 office 2007 以上版本的文档处理有问题,不知道最新版本是否解决了,我没有试过。 PDFParse 这个东西,我用过 .net 版本,对中文不支持,不知道 Java 版本是否支持。

    其实 .net 下完全不需要用这些开源解决方案来解决,因为微软提供了一个官方的解决方案,这个解决方案叫 IFilter,这个过滤器是为 SQL SERVER 的全文索引设计的,但第三方软件可以调用API来完成文档的提取工作。

    为了方便大家,我把 IFilter 转换的功能封装到了一个开源的组件中去,大家可以到下面地址去下载源码:HBTextParse .

    调用很简单:

    这个是提取文件中的文本到字符串的代码

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    if (openFileDialog.ShowDialog() == DialogResult.OK)
    {
        //要转换的文件
        textBoxFilePath.Text = openFileDialog.FileName;
         
        //实例化 TextParse ,传入要转换的文件路径
        TextParse textParse = new TextParse(textBoxFilePath.Text);
         
        //提取文件中的文本,并输出
        richTextBoxView.Text = textParse.ConvertToString();
    }
     
    这个是将文件转换为文本文件的代码:

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    if (saveFileDialog.ShowDialog() == DialogResult.OK)
    {
        try
        {
            //实例化 TextParse,传入要转换的文件的路径
            TextParse textParse = new TextParse(textBoxFilePath.Text);
             
            //将文件转换到 saveFileDialog.FileName 指定的文本文件中
            textParse.ConvertToFile(saveFileDialog.FileName);
        }
        catch (Exception ex)
        {
            MessageBox.Show(ex.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
        }
    }
     

    要注意的问题是提取 Pdf 文档,如果机器是 64为操作系统,必须要安装 Adobe PDF iFilter 9 for 64-bit platforms. 否则会报异常。这个问题我搞了将近一天才搞定。

    支持的文档类型:

    目前这个组件可以支持所有 Microsoft office 提供的文档类型,包括 *.rtf, *.doc, *.docx, *.xls, *.xlsx, *.ppt, *.pptx 等等

    除了微软Office的文档外,还可以转换html 文档:可以把html文档中的文本提取出来(不包含标签)Pdf 文档:我测试过,对中文支持没有问题

    Txt 文档
    这个代码的核心部分是一个叫 FilterCode 的类。这个类是从 http://ifilter.codeplex.com/ 这个地方下载的,我对这个类做了改进,加入了转换到文件的方法。我把这个类的代码贴出来,如果对如何调用IFilter的windows API感兴趣,可以参考这段代码

    IFilter 的相关API函数如下:通常这些API函数就可以通过IFilter接口提取文本。

       [DllImport("query.dll", SetLastError = true, CharSet = CharSet.Unicode)]
            static extern int LoadIFilter(string pwcsPath,
                      [MarshalAs(UnmanagedType.IUnknown)] object pUnkOuter,
                      ref IFilter ppIUnk);
    

     

            [ComImport, Guid("89BCB740-6119-101A-BCB7-00DD010655AF")]
            [InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
            public interface IFilter
            {
                /// <summary>
                /// The IFilter::Init method initializes a filtering session.
                /// </summary>
                [PreserveSig]
                IFilterReturnCodes Init(
                    //[in] Flag settings from the IFILTER_INIT enumeration for
                    // controlling text standardization, property output, embedding
                    // scope, and IFilter access patterns. 
                  IFILTER_INIT grfFlags,
    
                  // [in] The size of the attributes array. When nonzero, cAttributes
                    //  takes 
                    // precedence over attributes specified in grfFlags. If no
                    // attribute flags 
                    // are specified and cAttributes is zero, the default is given by
                    // the 
                    // PSGUID_STORAGE storage property set, which contains the date and
                    //  time 
                    // of the last write to the file, size, and so on; and by the
                    //  PID_STG_CONTENTS 
                    // 'contents' property, which maps to the main contents of the
                    // file. 
                    // For more information about properties and property sets, see
                    // Property Sets. 
                  int cAttributes,
    
                  //[in] Array of pointers to FULLPROPSPEC structures for the
                    // requested properties. 
                    // When cAttributes is nonzero, only the properties in aAttributes
                    // are returned. 
                  IntPtr aAttributes,
    
                  // [out] Information about additional properties available to the
                    //  caller; from the IFILTER_FLAGS enumeration. 
                  out IFILTER_FLAGS pdwFlags);
    
                /// <summary>
                /// The IFilter::GetChunk method positions the filter at the beginning
                /// of the next chunk, 
                /// or at the first chunk if this is the first call to the GetChunk
                /// method, and returns a description of the current chunk. 
                /// </summary>
                [PreserveSig]
                IFilterReturnCodes GetChunk(out STAT_CHUNK pStat);
    
                /// <summary>
                /// The IFilter::GetText method retrieves text (text-type properties)
                /// from the current chunk, 
                /// which must have a CHUNKSTATE enumeration value of CHUNK_TEXT.
                /// </summary>
                [PreserveSig]
                IFilterReturnCodes GetText(
                    // [in/out] On entry, the size of awcBuffer array in wide/Unicode
                    // characters. On exit, the number of Unicode characters written to
                    // awcBuffer. 
                    // Note that this value is not the number of bytes in the buffer. 
                    ref int pcwcBuffer,
    
                    // Text retrieved from the current chunk. Do not terminate the
                    // buffer with a character.  
                    [Out(), MarshalAs(UnmanagedType.LPWStr)] 
           StringBuilder awcBuffer);
    
                /// <summary>
                /// The IFilter::GetValue method retrieves a value (public
                /// value-type property) from a chunk, 
                /// which must have a CHUNKSTATE enumeration value of CHUNK_VALUE.
                /// </summary>
                [PreserveSig]
                IFilterReturnCodes GetValue(
                    // Allocate the PROPVARIANT structure with CoTaskMemAlloc. Some
                    // PROPVARIANT 
                    // structures contain pointers, which can be freed by calling the
                    // PropVariantClear function. 
                    // It is up to the caller of the GetValue method to call the
                    //   PropVariantClear method.            
                    // ref IntPtr ppPropValue
                    // [MarshalAs(UnmanagedType.Struct)]
                    ref IntPtr PropVal);
    
                /// <summary>
                /// The IFilter::BindRegion method retrieves an interface representing
                /// the specified portion of the object. 
                /// Currently reserved for future use.
                /// </summary>
                [PreserveSig]
                IFilterReturnCodes BindRegion(ref FILTERREGION origPos,
                  ref Guid riid, ref object ppunk);
            }

    从文档中提取文本的代码如下:

     

     /// <summary>
            /// Utilizes IFilter interface in Windows to parse the contents of files.
            /// </summary>
            /// <param name="path">Path - Location of file to parse</param>
            /// <param name="buffer">Buffer - Return text artifacts</param>
            /// <returns>Raw set of strings from the document in plain text format.</returns>
            public void GetTextFromDocument(string path, ref StringBuilder buffer)
            {
                IFilter filter = null;
                int hresult;
                IFilterReturnCodes rtn;
    
                // Initialize the return buffer to 64K.
                buffer = new StringBuilder(64 * 1024);
    
                // Try to load the filter for the path given.
                hresult = LoadIFilter(path, new IntPtr(0), ref filter);
                if (hresult == 0)
                {
                    IFILTER_FLAGS uflags;
    
                    // Init the filter provider.
                    rtn = filter.Init(
                            IFILTER_INIT.IFILTER_INIT_CANON_PARAGRAPHS |
                            IFILTER_INIT.IFILTER_INIT_CANON_HYPHENS |
                            IFILTER_INIT.IFILTER_INIT_CANON_SPACES |
                            IFILTER_INIT.IFILTER_INIT_APPLY_INDEX_ATTRIBUTES |
                            IFILTER_INIT.IFILTER_INIT_INDEXING_ONLY,
                            0, new IntPtr(0), out uflags);
                    if (rtn == IFilterReturnCodes.S_OK)
                    {
                        STAT_CHUNK statChunk;
    
                        // Outer loop will read chunks from the document at a time.  For those
                        // chunks that have text, the contents will be pulled and put into the
                        // return buffer.
                        bool bMoreChunks = true;
                        while (bMoreChunks)
                        {
                            rtn = filter.GetChunk(out statChunk);
                            if (rtn == IFilterReturnCodes.S_OK)
                            {
                                // Ignore all non-text chunks.
                                if (statChunk.flags != CHUNKSTATE.CHUNK_TEXT)
                                    continue;
    
                                // Check for white space items and add the appropriate breaks.
                                switch (statChunk.breakType)
                                {
                                    case CHUNK_BREAKTYPE.CHUNK_NO_BREAK:
                                        break;
    
                                    case CHUNK_BREAKTYPE.CHUNK_EOW:
                                        buffer.Append(' ');
                                        break;
    
                                    case CHUNK_BREAKTYPE.CHUNK_EOC:
                                    case CHUNK_BREAKTYPE.CHUNK_EOP:
                                    case CHUNK_BREAKTYPE.CHUNK_EOS:
                                        buffer.AppendLine();
                                        break;
                                }
    
                                // At this point we have a text chunk.  The following code will pull out
                                // all of it and add it to the buffer.
                                bool bMoreText = true;
                                while (bMoreText)
                                {
                                    // Create a temporary string buffer we can use for the parsing algorithm.
                                    int cBuffer = DefaultBufferSize;
                                    StringBuilder sbBuffer = new StringBuilder(DefaultBufferSize);
    
                                    // Read the next piece of data up to the size of our local buffer.
                                    rtn = filter.GetText(ref cBuffer, sbBuffer);
                                    if (rtn == IFilterReturnCodes.S_OK || rtn == IFilterReturnCodes.FILTER_S_LAST_TEXT)
                                    {
                                        // If any data was returned, scrub it and then add it to the buffer.
                                        CleanUpCharacters(cBuffer, sbBuffer);
                                        buffer.Append(sbBuffer.ToString());
    
                                        // If we got back some text but there is no more, terminate the loop.
                                        if (rtn == IFilterReturnCodes.FILTER_S_LAST_TEXT)
                                        {
                                            bMoreText = false;
                                            break;
                                        }
                                    }
                                    // Once all data is exhausted, we are done so terminate.
                                    else if (rtn == IFilterReturnCodes.FILTER_E_NO_MORE_TEXT)
                                    {
                                        bMoreText = false;
                                        break;
                                    }
                                    // Check for any fatal errors.  It is a bug if you land here.
                                    else if (rtn == IFilterReturnCodes.FILTER_E_NO_TEXT)
                                    {
                                        System.Diagnostics.Debug.Assert(false, "Should not get here");
                                        throw new InvalidOperationException();
                                    }
                                }
                            }
                            // Once all chunks have been read, we are done with the file.
                            else if (rtn == IFilterReturnCodes.FILTER_E_END_OF_CHUNKS)
                            {
                                bMoreChunks = false;
                                break;
                            }
                            else if (rtn == IFilterReturnCodes.FILTER_E_EMBEDDING_UNAVAILABLE ||
                                rtn == IFilterReturnCodes.FILTER_E_LINK_UNAVAILABLE)
                            {
                                continue;
                            }
                            else
                            {
                                throw new COMException("IFilter COM error: " + rtn.ToString());
                            }
                        }
                    }
                }
                else
                {
                    // If you get here there is no filter for the file type you asked for.  Throw an
                    // exception for the caller.
                    throw new InvalidOperationException("Failed to find IFilter for file " + path);
                }
            }